Building EC-Earth 4

This section will cover how the EC-Earth 4 components are built (i.e. compiled). It is assumed that the EC-Earth 4 source code has been cloned from the Git repository and the directory structure is created according to the previous section.

The ScriptEngine software is used to handle the build and runtime environment scripts and Ready2couple (rdy2cpl) for automatic creation of the OASIS coupler set-up. While it is possible to prepare Makefiles, configurations and run scripts manually, thus building and running EC-Earth 4 without using ScriptEngine, this is not covered in this documentation.

Prerequisites

In order to build and run EC-Earth 4, the following software must be available:

Python 3.8 or later
Anaconda or Mamba (not strictly necessary but strongly recommended and assumed hereafter)
Fortran, C, C++ compilers
MPI compiler wrappers, headers, modules and libraries
NetCDF headers, modules and libraries
ecCodes headers and libraries

Creating the Python virtual environment

ScriptEngine and it’s dependencies are written in Python and they are provided as Python packages on PyPI (installed via pip) and conda-forge (via conda or mamba). It is strongly recommended to install ScriptEngine from conda-forge because it simplifies the handling of dependencies a lot.

Note

Some HPCs require special install methods. For more information and alternative install methods, see “Machine-specific builds”.

The EC-Earth 4 source code comes with an environment file that can be used to create a conda environment and install all dependencies automatically. The environment a conda environment and install all dependencies automatically. The environment is created and activated like this:

ecearth4> conda env create --name ece4 --file environment.yml
ecearth4> conda activate ece4

This will create the conda environment in a default location for the user. Note that the environment needs to be created only once, but it has to be activated each time you open a new shell.

Alternatively, the conda environment can be created in a subdirectory of the EC-Earth 4 source code (or any other directory, as needed):

ecearth4> conda env create --prefix .conda --file environment.yml
ecearth4> conda activate ./.conda

In this example, the environment is created in the .conda subdirectory, but of course other names can be chosen.

Caution

Although the command to create the environment looks rather simple, a lot of things are happening behind the scenes. In particular, some compilation of MPI programs is needed to satisfy some of the dependencies. This means that a working MPI environment is needed at the time when the environment is created.

If the environment cannot be created correctly, a likely source of errors is the user environment. Make sure to check the platform definition files (in scripts/platforms) for information on loaded modules, etc.

You can test if the environment is correctly set up by running ScriptEngine:

ecearth4> se --help

The output should include in particular the rdy2cpl.make_all task, which ensures that Ready2couple was correctly installed.

Once the Python environment is created (once) and activated (once for every new shell), everything is ready for building the model.

Building the EC-Earth 4 components

This section will cover how the EC-Earth 4 components are built (i.e. compiled). It is assumed that the EC-Earth 4 source code has been checked out, ScriptEngine and Ready2couple are installed and the directory structure is created according to the previous section.

Overview

The source code of the EC-Earth 4 components resides in subdirectories of sources:

ecearth4> ls -1 sources
amip-forcing/
nemo-4.2/
oifs-48r1/
oasis3-mct-5.2/
runoff-mapper/
se/
util/
xios-2.5+/

We cover primarily the AMIP (atmosphere-only) and GCM (atmosphere-ocean) configurations of EC-Earth 4, therefore focusing on OpenIFS (oifs-48r1), NEMO (nemo-4.2), OASIS (oasis3-mct-5.2), XIOS (xios-2.5+), the Runoff-mapper (runoff-mapper) and the AMIP-Forcing-reader (amip-forcing).

The ScriptEngine scripts needed to built the components are located in the scripts/build subdirectory:

ecearth4> cd scripts/build
se> ls -1
platforms/
templates/
compile-amipfr.yml
compile-components.yml
compile-nemo.yml
compile-oasis.yml
compile-oifs.yml
compile-rnfm.yml
compile-xios.yml
user-settings.yml

The directory contains a number of YAML scripts, which are passed to ScriptEngine and will control the user and platform configuration, the preparation of makefiles and the like, and finally trigger the compilation of the components. Detailed information about how ScriptEngine works with YAML scripts can be found in the ScriptEngine documentation. For this documentation it suffices to mention that multiple scripts can be passed to the ScriptEngine se executable and are processed in order:

> se first.yml second.yml third.yml
2023-05-22 11:22:09 INFO [se.cli] Logging configured and started
2023-05-22 11:22:09 INFO [se.task:echo <cc24233403>] Hello from firs...
Hello from first script!
2023-05-22 11:22:09 INFO [se.task:echo <cbe525224f>] Hello from seco...
Hello from second script!
2023-05-22 11:22:09 INFO [se.task:echo <83069dab12>] Hello from thir...
Hello from third script!

For a brief overview of se options and arguments, run se --help.

Note

The actual names of all scripts are just convenient names. There is nothing special about them and they could be changed at will. Scripts could also be split or merged. As far as ScriptEngine is concerned, it will just process, in order, all scripts given as arguments.

The basic approach for building the EC-Earth 4 components, will be to call the scripts for user settings, the platform settings and the actual compilation commands in one go:

se> se user-settings.yml ../platforms/<MY_PLATFORM>.yml compile-<COMPONENT>.yml

Note

It is important to always include the user and platform settings scripts in every compilation! The settings are not magically stored between the ScriptEngine runs (other than in the scripts) and must always be included!

The order in which the EC-Earth 4 components must be compiled (due to inter-component dependencies) is

OASIS
XIOS
OpenIFS, NEMO, AMIP-Forcing-reader, Runoff-mapper (in any order)

Hint

There is also a compile-components.yml script, which compiles selected components in the correct order. The user and platform settings still have to be provided.

User settings

The user-settings.yml file is presently very simple and provides some basic configuration for the build process:

build:
  # -- Applies to all
  jobs: 10
  clean: True

  # -- Applies to OpenIFS
  oifs_exe:           # OIFS precision: ['SP-GCM','DP-GCM'] to build both SP and DP, defaults to ['DP-GCM']

  # -- Applies to NEMO
  oasis_active:       # NEMO coupled (default: True). Set to `False` for STANDALONE nemo.
  top_active:         # NEMO with TOP (biochemistry/inert tracers). Defaults to `False`.
  isf_active:         # Make option to open ice shelves available. Defaults to `False`.
  rebuild_nemo: False # Utility to create global files

  # -- Components list for 'compile-components.yml'
  # OGCM: [oasis, xios, nemo]
  # AGCM: [oasis, xios, openifs, amipfr]
  # AOGCM (default): [oasis, xios, oifs, nemo, rnfm, amipfr]
  components:

where the parameters are:

experiment.base_dir : the top directory of the EC-Earth 4 code. Here it is set relatively to the current working directory se.cli.cwd.
experiment.src_dir : the location of the sources directory. Here it is set relatively to the experiment.base_dir
build.jobs : number of parallel process to build components, when possible
build.clean : if true, the build process will start from scratch (equivalent to make clean)
build.oifs_exe : Build OIFS in``SP-GCM`` (single precision) and DP-GCM (double precision). If empty or undefined, only DP-GCM is built.
build.oasis_active : switch to build NEMO for standalone (False) or coupled-to-OpenIFS (True) experiments
build.top_active : switch to build NEMO with (True) or without (False) TOP. TOP handles passive tracers in NEMO, and is required by PISCES and/or for CFCs
build.isf_active : switch to build NEMO with (True) or without (False) ice-shelf functionality.
build.components : list of components to compile. Used ONLY with compile-components.yml. See Build with compile-components.yml hereafter

Note that NEMO and/or XIOS build directories will differ according to the two oasis_active and top_active parameters, allowing for multiple NEMO configurations to be built inside the same source directory. The four possible NEMO configurations are:

COUPLED (default) – Coupled with OASIS but without passive tracers.
COUPLED-TOP – COUPLED with passive tracers (e.g. PISCES is activated).
STANDALONE – Ocean-only runs, no passive tracers.
STANDALONE-TOP – Ocean-only with passive tracers (e.g. water mass age, pisces,…)

When oasis_active and top_active parameters are empty or undefined, the COUPLED configurations of NEMO and XIOS components are built.

Note to developers: xios is built in its build_ecearth dir (for ocean-atmosphere coupled) or build_no-oasis dir (for nemo-only).

Build with `compile-components.yml`

If you compile several components at once with compile-components.yml, the build.components list is used. Typically, this list will match your target run configuration, i.e. model_config.components (see Running simple experiments). Note that build.oasis_active is ignored by compile-components.yml and is instead inferred from the list of components. Examples of possible lists and their corresponding configurations:

[xios, nemo] : Ocean-only
[oasis, xios, oifs, amipfr] : Atmosphere-only
[oasis, xios, oifs, nemo, rnfm] : Coupled Atmosphere-Ocean
[oasis, xios, oifs, nemo, rnfm, amipfr] : this is the default list (if empty or undefined). All components are compiled, allowing for both coupled Atmosphere-Ocean and Atmosphere-only experiments, but not ocean-only ones!

Hint

The syntax of Python, YAML, Markdown and some other modern languages and formats is based on indentation. Tab characters can not provide consistent indentation, and it has therefore become a major offence to use tab characters in source code! Make sure that your editor replaces tab characters with spaces and that you follow the indentation convention (i.e. how many spaces per level) of the source code you are working with.

Again, for details regarding the specific syntax used in the configuration definition, refer to the ScriptEngine documentation, particularly the Writing Scripts section therein.

Platform settings

The settings for different platforms are organised in the platforms/ subdirectory. The platform configuration is naturally a bit heavier than the user settings and comprises things like compiler and library settings, preprocessor macros and other details.

Given that there are many aspects and details of platform configuration, the information is structured using a tree-like organisation of configuration parameters. In the configuration scripts for ScriptEngine, this makes use of YAML dictionaries. For example, this is how the compiler-related configuration parameters could be defined for a particular platform:

base.context:
  build:
    lang:
      fortran:
        compiler: mpif90
        flags: -g -O2 -fdefault-real-8 -fdefault-double-8 -ffree-line-length-none
        preprocessor: !noparse "{{build.lang.c.preprocessor}}"
      c:
        compiler: mpicc
        flags: -g
        preprocessor: cpp
      linker:
        command: !noparse "{{build.lang.fortran.compiler}}"
      make:
        command: gmake

As another example, this section to configure some needed libraries could look like:

base.context:
  build:
    # [...]
    deps:
      mpi: null
      lapack:
        libs: [openblas]
      grib:
        base_dir: /software/sse/manual/eccodes/2.8.2/nsc1-gcc-2018a-eb
        inc_dir: !noparse "{{build.deps.grib.base_dir}}/include"
        lib_dir: !noparse "{{build.deps.grib.base_dir}}/lib"
        libs: [eccodes_f90, eccodes, pthread]
        samples_path: !noparse "{{build.deps.grib.base_dir}}/share/eccodes/samples:\
          {{build.deps.grib.base_dir}}/share/eccodes/ifs_samples/grib1_mlgrib2"
      netcdf:
        mod_dir: /software/sse/manual/netcdf/4.4.1.1/nsc1-gcc-2018a-eb/include
        libs: [netcdff, netcdf]
      hdf5: null

Again, the configuration parameters are available for later use in a structured way, such as {{build.deps.grib.lib_dir}}.

Some details are worth further explanation in the above example: The syntax mpi: null and hdf5: null is used in YAML to denote an empty value. Thus, there is no special MPI or HDF5 configuration on this platform (the configuration is provided by modules in this case).

Second, the *.libs parameters must be defined as YAML lists (with brackets and commas), even if there is only one library.

Hint

If you want to test user or platform setting scripts, it is possible to run these with ScriptEngine without actual compilation:

> se user-settings.yml platforms/<MY_PLATFORM>.yml

i.e. just omit the compile script(s). Thus, you can make sure the setting scripts are syntactically correct before you attempt the actual compilation.

Generally, it will be easiest to start from a known-to-work platform file, copy and adapt to the platform.

Completing the environment

Once the EC-Earth 4 components are built, there is one more step before the environment is complete and ready to use for experiments (unless you only intend to run NEMO STANDALONE without OASIS). In order to create grid specific files for the OASIS coupler, EC-Earth 4 uses ready2couple (rdy2cpl), which generates grid description files and regridding weights on-the-fly, according to the grids chosen in the run-time configuration.

In order for ready2couple to work, it must be configured to find the OASIS libraries compiled in the step above. Two environment variables must be set/modified for this:

OASIS_BUILD_PATH must point to the OASIS build directory
the LD_LIBRARY_PATH variable must include ${OASIS_BUILD_PATH}/lib

The OASIS_BUILD_PATH will usually be something like <YOUR_ECE4_PATH>/sources/oasis3-mct-5.2/arch_ecearth.

The correct setting of these variables can be tested by running, with the EC-Earth 4 virtual environment activated, the following command:

> r2c --help

which should produce no error message.

Note

The OASIS installation process for pyOASIS suggests setting further environment variables. These variables are not needed for ready2couple.

Machine-specific builds

An alternative way to build the Python environment is as a virtual environment

python3.9 -m venv .se
source .se/bin/activate

pip install -U pip

pip install scriptengine
pip install scriptengine-tasks-hpc

pip install git+https://github.com/ecmwf/ecbundle

# After installing OASIS
pip install git+https://github.com/uwefladrich/rdy2cpl

Note

The above method is neither recommended nor supported by the EC-Earth team and should only be used if building the conda environment above is not possible.

For OIFS to run on Puhti (CSC) with GNU, you need edit sources/oifs-48r1/ifs-source/cmake/compile_flags.cmake and comment out the -ffpe-trap=-part.

In case you compile with Intel-MPI, do the following before installing ready2cpl:

export MPICC=${I_MPI_ROOT}/bin/mpicc
export MPILD=””
export MPI4PY_BUILD_CONFIGURE=""
pip install --no-cache-dir mpi4py

# Tests should be run through srun:

srun -n 1 r2c --help
srun -n 6 r2c --couple-grid-spec ece_couple_grids.yml namcouple.tmp.yml

Building EC-Earth 4

Prerequisites

Creating the Python virtual environment

Building the EC-Earth 4 components

Overview

User settings

Build with compile-components.yml

Platform settings

Completing the environment

Machine-specific builds

Building EC-Earth 4

Building the EC-Earth 4 components

Build with `compile-components.yml`