Building EC-Earth 4

This section will cover how the EC-Earth 4 components are built (i.e. compiled). It is assumed that the EC-Earth 4 source code has been cloned from the Git repository and the directory structure is created according to the previous section.

The ScriptEngine software is used to handle the build and runtime environment scripts and Ready2couple (rdy2cpl) for automatic creation of the OASIS coupler set-up. While it is possible to prepare Makefiles, configurations and run scripts manually, thus building and running EC-Earth 4 without using ScriptEngine, this is not covered in this documentation.

Prerequisites

In order to build and run EC-Earth 4, the following software must be available:

  • Python 3.8 or later

  • Anaconda or Mamba (not strictly necessary but strongly recommended and assumed hereafter)

  • Fortran, C, C++ compilers

  • MPI compiler wrappers, headers, modules and libraries

  • NetCDF headers, modules and libraries

  • ecCodes headers and libraries

Creating the Python virtual environment

ScriptEngine and it’s dependencies are written in Python and they are provided as Python packages on PyPI (installed via pip) and conda-forge (via conda or mamba). It is strongly recommended to install ScriptEngine from conda-forge because it simplifies the handling of dependencies a lot.

The EC-Earth 4 source code comes with an environment file that can be used to create a conda environment and install all dependencies automatically. The environment a conda environment and install all dependencies automatically. The environment is created and activated like this:

ecearth4> conda env create --name ece4 --file environment.yml
ecearth4> conda activate ece4

This will create the conda environment in a default location for the user. Note that the environment needs to be created only once, but it has to be activated each time you open a new shell.

Alternatively, the conda environment can be created in a subdirectory of the EC-Earth 4 source code (or any other directory, as needed):

ecearth4> conda env create --prefix .conda --file environment.yml
ecearth4> conda activate ./.conda

In this example, the environment is created in the .conda subdirectory, but of course other names can be chosen.

Caution

Although the command to create the environment looks rather simple, a lot of things are happening behind the scenes. In particular, some compilation of MPI programs is needed to satisfy some of the dependencies. This means that a working MPI environment is needed at the time when the environment is created.

If the environment cannot be created correctly, a likely source of errors is the user environment. Make sure to check the platform definition files (in sources/se/platforms) for information on loaded modules, etc.

You can test if the environment is correctly set up by running ScriptEngine:

ecearth4> se --help

The output should include in particular the rdy2cpl.make_all task, which ensures that Ready2couple was correctly installed.

Once the Python environment is created (once) and activated (once for every new shell), everything is ready for building the model.

Building the EC-Earth 4 components

This section will cover how the EC-Earth 4 components are built (i.e. compiled). It is assumed that the EC-Earth 4 source code has been checked out, ScriptEngine and Ready2couple are installed and the directory structure is created according to the previous section.

Overview

The source code of the EC-Earth 4 components resides in subdirectories of sources:

ecearth4> ls -1 sources
amip-forcing/
nemo-4.2/
oasis3-mct-5.0/
oifs-43r3v2/
runoff-mapper/
se/
util/
xios-2.5+/

We cover primarily the AMIP (atmosphere-only) and GCM (atmosphere-ocean) configurations of EC-Earth 4, therefore focusing on OpenIFS (oifs-43r3v2), NEMO (nemo-4.2), OASIS (oasis3-mct-5.0), XIOS (xios-2.5+), the Runoff-mapper (runoff-mapper) and the AMIP-Forcing-reader (amip-forcing).

The ScriptEngine scripts needed to built the components are located in the se subdirectory:

ecearth4> cd sources/se
se> ls -1
platforms/
templates/
compile-all.yml
compile-amipfr.yml
compile-nemo.yml
compile-oasis.yml
compile-oifs.yml
compile-rnfm.yml
compile-xios.yml
user-settings.yml

The directory contains a number of YAML scripts, which are passed to ScriptEngine and will control the user and platform configuration, the preparation of makefiles and the like, and finally trigger the compilation of the components. Detailed information about how ScriptEngine works with YAML scripts can be found in the ScriptEngine documentation. For this documentation it suffices to mention that multiple scripts can be passed to the ScriptEngine se executable and are processed in order:

> se first.yml second.yml third.yml
2023-05-22 11:22:09 INFO [se.cli] Logging configured and started
2023-05-22 11:22:09 INFO [se.task:echo <cc24233403>] Hello from firs...
Hello from first script!
2023-05-22 11:22:09 INFO [se.task:echo <cbe525224f>] Hello from seco...
Hello from second script!
2023-05-22 11:22:09 INFO [se.task:echo <83069dab12>] Hello from thir...
Hello from third script!

For a brief overview of se options and arguments, run se --help.

Note

The actual names of all scripts are just convenient names. There is nothing special about them and they could be changed at will. Scripts could also be split or merged. As far as ScriptEngine is concerned, it will just process, in order, all scripts given as arguments.

The basic approach for building the EC-Earth 4 components, will be to call the scripts for user settings, the platform settings and the actual compilation commands in one go:

se> se user-settings.yml platforms/<MY_PLATFORM>.yml compile-<COMPONENT>.yml

Note

It is important to always include the user and platform settings scripts in every compilation! The settings are not magically stored between the ScriptEngine runs (other than in the scripts) and must always be included!

The order in which the EC-Earth 4 components must be compiled (due to inter-component dependencies) is

  • OASIS

  • XIOS

  • OpenIFS, NEMO, AMIP-Forcing-reader, Runoff-mapper (in any order)

Hint

There is also a compile-all.yml script, which compiles all components in the correct order. The user and platform settings still have to be provided.

User settings

The user-settings.yml file is presently very simple and provides some basic configuration for the build process:

# User-dependent configuration for EC-Earth 4 build
base.context:
  experiment:
    base_dir: "{{se.cli.cwd}}/../.."
    src_dir: !noparse "{{experiment.base_dir}}/sources"
  build:
    jobs: 10
    clean: true

If set as in the default shown above, the EC-Earth 4 source code is assumed relative to the current working directory (se.cli.cwd). The building of components will use parallel make (where available) with 10 processes. Finally, the build process will “clean up” (the equivalent of make clean where available) If set as in the default shown above, the EC-Earth 4 source code is assumed relative to the current working directory (se.cli.cwd). The building of components will use parallel make (where available) with 10 processes. Finally, the build process will “clean up” (the equivalent of make clean where available) before compiling.

Hint

The syntax of Python, YAML, Markdown and some other modern languages and formats is based on indentation. Tab characters can not provide consistent indentation, and it has therefore become a major offence to use tab characters in source code! Make sure that your editor replaces tab characters with spaces and that you follow the indentation convention (i.e. how many spaces per level) of the source code you are working with.

Again, for details regarding the specific syntax used in the configuration definition, refer to the ScriptEngine documentation, particularly the Writing Scripts section therein.

For the course of this documentation, it is important that the main.base_dir configuration parameter points to the correct directory.

Platform settings

The settings for different platforms are organised in the platforms/ subdirectory. The platform configuration is naturally a bit heavier than the user settings and comprises things like compiler and library settings, preprocessor macros and other details.

Given that there are many aspects and details of platform configuration, the information is structured using a tree-like organisation of configuration parameters. In the configuration scripts for ScriptEngine, this makes use of YAML dictionaries. For example, this is how the compiler-related configuration parameters could be defined for a particular platform:

base.context:
  build:
    lang:
      fortran:
        compiler: mpif90
        flags: -g -O2 -fdefault-real-8 -fdefault-double-8 -ffree-line-length-none
        preprocessor: !noparse "{{build.lang.c.preprocessor}}"
      c:
        compiler: mpicc
        flags: -g
        preprocessor: cpp
      linker:
        command: !noparse "{{build.lang.fortran.compiler}}"
      make:
        command: gmake

As another example, this section to configure some needed libraries could look like:

base.context:
  build:
    # [...]
    deps:
      mpi: null
      lapack:
        libs: [openblas]
      grib:
        base_dir: /software/sse/manual/eccodes/2.8.2/nsc1-gcc-2018a-eb
        inc_dir: !noparse "{{build.deps.grib.base_dir}}/include"
        lib_dir: !noparse "{{build.deps.grib.base_dir}}/lib"
        libs: [eccodes_f90, eccodes, pthread]
        samples_path: !noparse "{{build.deps.grib.base_dir}}/share/eccodes/samples:\
          {{build.deps.grib.base_dir}}/share/eccodes/ifs_samples/grib1_mlgrib2"
      netcdf:
        mod_dir: /software/sse/manual/netcdf/4.4.1.1/nsc1-gcc-2018a-eb/include
        libs: [netcdff, netcdf]
      hdf5: null

Again, the configuration parameters are available for later use in a structured way, such as {{build.deps.grib.lib_dir}}.

Some details are worth further explanation in the above example: The syntax mpi: null and hdf5: null is used in YAML to denote an empty value. Thus, there is no special MPI or HDF5 configuration on this platform (the configuration is provided by modules in this case).

Second, the *.libs parameters must be defined as YAML lists (with brackets and commas), even if there is only one library.

Hint

If you want to test user or platform setting scripts, it is possible to run these with ScriptEngine without actual compilation:

> se user-settings.yml platforms/<MY_PLATFORM>.yml

i.e. just omit the compile script(s). Thus, you can make sure the setting scripts are syntactically correct before you attempt the actual compilation.

Generally, it will be easiest to start from a known-to-work platform file, copy and adapt to the platform.

Completing the environment

Once the EC-Earth 4 components are built, there is one more step before the environment is complete and ready to use for experiments. In order to create grid specific files for the OASIS coupler, EC-Earth 4 uses ready2couple (rdy2cpl), which generates grid description files and regridding weights on-the-fly, according to the grids chosen in the run-time configuration.

In order for ready2couple to work, it must be configured to find the OASIS libraries compiled in the step above. Two environment variables must be set/modified for this:

  • OASIS_BUILD_PATH must point to the OASIS build directory

  • the LD_LIBRARY_PATH variable must include ${OASIS_BUILD_PATH}/lib

The OASIS_BUILD_PATH will usually be something like <YOUR_ECE4_PATH>/sources/oasis3-mct-5.2/arch_ecearth.

The correct setting of these variables can be tested by running, with the EC-Earth 4 virtual environment activated, the following command:

> r2c --help

which should produce no error message.

Note

The OASIS installation process for pyOASIS suggests setting further environment variables. These variables are not needed for ready2couple.