Additional information

Known issues

Bit-reproducibility:

Note

EC-Earth will generally not be bit-for-bit reproducible when the number of MPI tasks is varied. If a user wishes to run an experiment which is bit-identical to a previous experiment the number of MPI tasks should not be changed.

As a general rule, climate models do not yield bit-identical results when an experiment is repeated after changing

  • hardware, e.g. moving to a new HPC or following a hardware upgrade

  • compilers, e.g. switching from GNU to Intel compilers

  • MPI implementation, e.g. switching from OpenMPI to IntelMPI

  • versions of libraries such as netCDF, BLAS or Intel MKL

However, it is not expected that such changes should result in significant changes in the model climate.

It has been noted that EC-Earth v4.1.0 and above may not produce bit-reproducible results when the following parameters change

  • Number of MPI tasks for OpenIFS (in coupled simulations)

  • Frequency of FullPos calls in OpenIFS (sample_rate in config_oifs.yml)

although this is dependent on the hardware and software used.

When the number of MPI tasks is changed, the order in which OASIS computes global sums can change and break bit-reproducibility. The bfb option in OASIS can overcome this issue but at a large and often unreasonable computer cost.

Changing the sampling frequency in OpenIFS, i.e. how often XIOS sends model fields to XIOS for writing, can break bit-reproducibility. This seems to be related to the math libraries (Intel MKL, OpenBLAS etc). Tests at HPC2020 (ECMWF) found that it is possible to have bit-reproducible results for different sample_rate as long as one of the following applies (but not guaranteed):

  • Have Intel CPUs and Intel MKL with export MKL_CBWR=AUTO,STRICT

  • Have AMD CPUs (HPC2020, NSC), and Intel MKL 19.0.5 or older, and export MKL_DEBUG_CPU=5 and export MKL_CBWR=AUTO,STRICT

  • Have any CPUs with OpenBLAS but build for SandyBridge architecture.

EC-Earth 4.1.0 is bit-reproducible with changing sample_rate on LUMI without any of the above settings, but setting export LIBSCI_ARCH_OVERRIDE=ivybridge (or sandybridge) is required for bit-identical results when the number of MPI processes.

Differences in OpenIFS 48r1

A new cycle of the OpenIFS atmosphere model (48r1) is introduced in EC-Earth v4.1.0. Here follows a list of changes to pay attention to:

  1. Initial data, ece-4-data-base, has changed as 48r1 needs different files than older versions. Make sure you have the latest data.

  2. Snow now has multiple layers and thus several snow fields are 3D.

  3. Many necessary libraries, e.g. ecCodes, are downloaded to sources/oifs-48r1/external and compiled with the model. Compile times may thus be longer.