Building |ece4| =================== This section will cover how the |ece4| components are built (i.e. compiled). It is assumed that the |ece4| source code has been cloned from the Git repository and the directory structure is created according to the previous section. The `ScriptEngine `_ software is used to handle the build and runtime environment scripts and `Ready2couple (rdy2cpl) `_ for automatic creation of the OASIS coupler set-up. While it is possible to prepare Makefiles, configurations and run scripts manually, thus building and running |ece4| without using ScriptEngine, this is not covered in this documentation. Prerequisites ------------- In order to build and run |ece4|, the following software must be available: * Python 3.8 or later * Anaconda or Mamba (not strictly necessary but strongly recommended and assumed hereafter) * Fortran, C, C++ compilers * MPI compiler wrappers, headers, modules and libraries * NetCDF headers, modules and libraries * `ecCodes `_ headers and libraries .. _creating_virtual_environment: Creating the Python virtual environment --------------------------------------- ScriptEngine and it's dependencies are written in Python and they are provided as Python packages on `PyPI `_ (installed via pip) and `conda-forge `_ (via conda or mamba). It is strongly recommended to install ScriptEngine from conda-forge because it simplifies the handling of dependencies a lot. The |ece4| source code comes with an environment file that can be used to create a conda environment and install all dependencies automatically. The environment a conda environment and install all dependencies automatically. The environment is created and activated like this: .. code-block:: shell ecearth4> conda env create --name ece4 --file environment.yml ecearth4> conda activate ece4 This will create the conda environment in a default location for the user. Note that the environment needs to be created only once, but it has to be activated each time you open a new shell. Alternatively, the conda environment can be created in a subdirectory of the |ece4| source code (or any other directory, as needed): .. code-block:: shell ecearth4> conda env create --prefix .conda --file environment.yml ecearth4> conda activate ./.conda In this example, the environment is created in the ``.conda`` subdirectory, but of course other names can be chosen. .. caution:: Although the command to create the environment looks rather simple, a lot of things are happening behind the scenes. In particular, some compilation of MPI programs is needed to satisfy some of the dependencies. This means that a working MPI environment is needed at the time when the environment is created. If the environment cannot be created correctly, a likely source of errors is the user environment. Make sure to check the platform definition files (in ``sources/se/platforms``) for information on loaded modules, etc. You can test if the environment is correctly set up by running ScriptEngine: .. code-block:: shell ecearth4> se --help The output should include in particular the ``rdy2cpl.make_all`` task, which ensures that Ready2couple was correctly installed. Once the Python environment is created (once) and activated (once for every new shell), everything is ready for building the model. Building the |ece4| components ---------------------------------- This section will cover how the |ece4| components are built (i.e. compiled). It is assumed that the |ece4| source code has been checked out, ScriptEngine and Ready2couple are installed and the directory structure is created according to the previous section. Overview ~~~~~~~~ The source code of the |ece4| components resides in subdirectories of ``sources``: .. code-block:: shell ecearth4> ls -1 sources amip-forcing/ nemo-4.2/ oasis3-mct-5.0/ oifs-43r3v2/ runoff-mapper/ se/ util/ xios-2.5+/ We cover primarily the AMIP (atmosphere-only) and GCM (atmosphere-ocean) configurations of |ece4|, therefore focusing on OpenIFS (``oifs-43r3v2``), NEMO (``nemo-4.2``), OASIS (``oasis3-mct-5.0``), XIOS (``xios-2.5+``), the Runoff-mapper (``runoff-mapper``) and the AMIP-Forcing-reader (``amip-forcing``). The ScriptEngine scripts needed to built the components are located in the ``se`` subdirectory: .. code-block:: shell ecearth4> cd sources/se se> ls -1 platforms/ templates/ compile-all.yml compile-amipfr.yml compile-nemo.yml compile-oasis.yml compile-oifs.yml compile-rnfm.yml compile-xios.yml user-settings.yml The directory contains a number of YAML scripts, which are passed to ScriptEngine and will control the user and platform configuration, the preparation of makefiles and the like, and finally trigger the compilation of the components. Detailed information about how ScriptEngine works with YAML scripts can be found in the `ScriptEngine documentation `_. For this documentation it suffices to mention that multiple scripts can be passed to the ScriptEngine ``se`` executable and are processed in order: .. code-block:: shell > se first.yml second.yml third.yml 2023-05-22 11:22:09 INFO [se.cli] Logging configured and started 2023-05-22 11:22:09 INFO [se.task:echo ] Hello from firs... Hello from first script! 2023-05-22 11:22:09 INFO [se.task:echo ] Hello from seco... Hello from second script! 2023-05-22 11:22:09 INFO [se.task:echo <83069dab12>] Hello from thir... Hello from third script! For a brief overview of ``se`` options and arguments, run ``se --help``. .. note:: The actual names of all scripts are just convenient names. There is nothing special about them and they could be changed at will. Scripts could also be split or merged. As far as ScriptEngine is concerned, it will just process, in order, all scripts given as arguments. The basic approach for building the |ece4| components, will be to call the scripts for user settings, the platform settings and the actual compilation commands in one go: .. code-block:: shell se> se user-settings.yml platforms/.yml compile-.yml .. note:: It is important to always include the user and platform settings scripts in every compilation! The settings are not magically stored between the ScriptEngine runs (other than in the scripts) and must always be included! The order in which the |ece4| components must be compiled (due to inter-component dependencies) is * OASIS * XIOS * OpenIFS, NEMO, AMIP-Forcing-reader, Runoff-mapper (in any order) .. hint:: There is also a ``compile-all.yml`` script, which compiles all components in the correct order. The user and platform settings still have to be provided. User settings ~~~~~~~~~~~~~ The ``user-settings.yml`` file is presently very simple and provides some basic configuration for the build process: .. code-block:: yaml+jinja # User-dependent configuration for EC-Earth 4 build base.context: experiment: base_dir: "{{se.cli.cwd}}/../.." src_dir: !noparse "{{experiment.base_dir}}/sources" build: jobs: 10 clean: true If set as in the default shown above, the |ece4| source code is assumed relative to the current working directory (``se.cli.cwd``). The building of components will use parallel make (where available) with 10 processes. Finally, the build process will "clean up" (the equivalent of ``make clean`` where available) If set as in the default shown above, the |ece4| source code is assumed relative to the current working directory (``se.cli.cwd``). The building of components will use parallel make (where available) with 10 processes. Finally, the build process will "clean up" (the equivalent of ``make clean`` where available) before compiling. .. hint:: The syntax of Python, YAML, Markdown and some other modern languages and formats is based on indentation. Tab characters can not provide consistent indentation, and it has therefore become a major offence to use tab characters in source code! Make sure that your editor replaces tab characters with spaces and that you follow the indentation convention (i.e. how many spaces per level) of the source code you are working with. Again, for details regarding the specific syntax used in the configuration definition, refer to the `ScriptEngine documentation `_, particularly the `Writing Scripts `_ section therein. For the course of this documentation, it is important that the ``main.base_dir`` configuration parameter points to the correct directory. Platform settings ~~~~~~~~~~~~~~~~~ The settings for different platforms are organised in the ``platforms/`` subdirectory. The platform configuration is naturally a bit heavier than the user settings and comprises things like compiler and library settings, preprocessor macros and other details. Given that there are many aspects and details of platform configuration, the information is structured using a tree-like organisation of configuration parameters. In the configuration scripts for ScriptEngine, this makes use of YAML dictionaries. For example, this is how the compiler-related configuration parameters could be defined for a particular platform: .. code-block:: yaml+jinja base.context: build: lang: fortran: compiler: mpif90 flags: -g -O2 -fdefault-real-8 -fdefault-double-8 -ffree-line-length-none preprocessor: !noparse "{{build.lang.c.preprocessor}}" c: compiler: mpicc flags: -g preprocessor: cpp linker: command: !noparse "{{build.lang.fortran.compiler}}" make: command: gmake As another example, this section to configure some needed libraries could look like: .. code-block:: yaml+jinja base.context: build: # [...] deps: mpi: null lapack: libs: [openblas] grib: base_dir: /software/sse/manual/eccodes/2.8.2/nsc1-gcc-2018a-eb inc_dir: !noparse "{{build.deps.grib.base_dir}}/include" lib_dir: !noparse "{{build.deps.grib.base_dir}}/lib" libs: [eccodes_f90, eccodes, pthread] samples_path: !noparse "{{build.deps.grib.base_dir}}/share/eccodes/samples:\ {{build.deps.grib.base_dir}}/share/eccodes/ifs_samples/grib1_mlgrib2" netcdf: mod_dir: /software/sse/manual/netcdf/4.4.1.1/nsc1-gcc-2018a-eb/include libs: [netcdff, netcdf] hdf5: null Again, the configuration parameters are available for later use in a structured way, such as ``{{build.deps.grib.lib_dir}}``. Some details are worth further explanation in the above example: The syntax ``mpi: null`` and ``hdf5: null`` is used in YAML to denote an empty value. Thus, there is no special MPI or HDF5 configuration on this platform (the configuration is provided by modules in this case). Second, the ``*.libs`` parameters must be defined as YAML lists (with brackets and commas), even if there is only one library. .. hint:: If you want to test user or platform setting scripts, it is possible to run these with ScriptEngine without actual compilation: ``> se user-settings.yml platforms/.yml`` i.e. just omit the compile script(s). Thus, you can make sure the setting scripts are syntactically correct before you attempt the actual compilation. Generally, it will be easiest to start from a known-to-work platform file, copy and adapt to the platform. .. _completing-the-environment: Completing the environment -------------------------- Once the |ece4| components are built, there is one more step before the environment is complete and ready to use for experiments. In order to create grid specific files for the OASIS coupler, |ece4| uses `ready2couple (rdy2cpl) `_, which generates grid description files and regridding weights on-the-fly, according to the grids chosen in the run-time configuration. In order for ready2couple to work, it must be configured to find the OASIS libraries compiled in the step above. Two environment variables must be set/modified for this: - ``OASIS_BUILD_PATH`` must point to the OASIS build directory - the ``LD_LIBRARY_PATH`` variable must include ``${OASIS_BUILD_PATH}/lib`` The ``OASIS_BUILD_PATH`` will usually be something like ``/sources/oasis3-mct-5.2/arch_ecearth``. The correct setting of these variables can be tested by running, with the |ece4| virtual environment activated, the following command: .. code-block:: shell > r2c --help which should produce no error message. .. note:: The OASIS installation process for pyOASIS suggests setting further environment variables. These variables are not needed for ready2couple.