Building |ece4| =================== This section will cover how the |ece4| components are built (i.e. compiled). It is assumed that the |ece4| source code has been cloned from the Git repository and the directory structure is created according to the previous section. The `ScriptEngine `_ software is used to handle the build and runtime environment scripts and `Ready2couple (rdy2cpl) `_ for automatic creation of the OASIS coupler set-up. While it is possible to prepare Makefiles, configurations and run scripts manually, thus building and running |ece4| without using ScriptEngine, this is not covered in this documentation. Prerequisites ------------- In order to build and run |ece4|, the following software must be available: * Python 3.8 or later * Anaconda or Mamba (not strictly necessary but strongly recommended and assumed hereafter) * Fortran, C, C++ compilers * MPI compiler wrappers, headers, modules and libraries * NetCDF headers, modules and libraries * `ecCodes `_ headers and libraries .. _creating_virtual_environment: Creating the Python virtual environment --------------------------------------- ScriptEngine and it's dependencies are written in Python and they are provided as Python packages on `PyPI `_ (installed via pip) and `conda-forge `_ (via conda or mamba). It is strongly recommended to install ScriptEngine from conda-forge because it simplifies the handling of dependencies a lot. .. note:: Some HPCs require special install methods. For more information and alternative install methods, see "Machine-specific builds". The |ece4| source code comes with an environment file that can be used to create a conda environment and install all dependencies automatically. The environment a conda environment and install all dependencies automatically. The environment is created and activated like this: .. code-block:: shell ecearth4> conda env create --name ece4 --file environment.yml ecearth4> conda activate ece4 This will create the conda environment in a default location for the user. Note that the environment needs to be created only once, but it has to be activated each time you open a new shell. Alternatively, the conda environment can be created in a subdirectory of the |ece4| source code (or any other directory, as needed): .. code-block:: shell ecearth4> conda env create --prefix .conda --file environment.yml ecearth4> conda activate ./.conda In this example, the environment is created in the ``.conda`` subdirectory, but of course other names can be chosen. .. caution:: Although the command to create the environment looks rather simple, a lot of things are happening behind the scenes. In particular, some compilation of MPI programs is needed to satisfy some of the dependencies. This means that a working MPI environment is needed at the time when the environment is created. If the environment cannot be created correctly, a likely source of errors is the user environment. Make sure to check the platform definition files (in ``scripts/platforms``) for information on loaded modules, etc. You can test if the environment is correctly set up by running ScriptEngine: .. code-block:: shell ecearth4> se --help The output should include in particular the ``rdy2cpl.make_all`` task, which ensures that Ready2couple was correctly installed. Once the Python environment is created (once) and activated (once for every new shell), everything is ready for building the model. .. _building_components: Building the |ece4| components ---------------------------------- This section will cover how the |ece4| components are built (i.e. compiled). It is assumed that the |ece4| source code has been checked out, ScriptEngine and Ready2couple are installed and the directory structure is created according to the previous section. Overview ~~~~~~~~ The source code of the |ece4| components resides in subdirectories of ``sources``: .. code-block:: shell ecearth4> ls -1 sources amip-forcing/ lpjg-4.1/ nemo-4.2/ oasis3-mct-5.2/ oifs-48r1/ runoff-mapper/ xios-3.0/ We cover primarily the AMIP (atmosphere-only) and GCM (atmosphere-ocean) configurations of |ece4|, therefore focusing on OpenIFS (``oifs-48r1``), NEMO (``nemo-4.2``), LPJ-Guess (``lpjg-4.1``), OASIS (``oasis3-mct-5.2``), XIOS (``xios-3.0``), the Runoff-mapper (``runoff-mapper``) and the AMIP-Forcing-reader (``amip-forcing``). The ScriptEngine scripts needed to built the components are located in the ``scripts/build`` subdirectory: .. code-block:: shell ecearth4> cd scripts/build se> ls -1 compile-amipfr.yml compile-components.yml compile-lpjg.yml compile-nemo.yml compile-oasis.yml compile-oifs.yml compile-rnfm.yml compile-xios.yml docker-build.yml templates/ user-settings.yml The directory contains a number of YAML scripts, which are passed to ScriptEngine and will control the user and platform configuration, the preparation of makefiles and the like, and finally trigger the compilation of the components. Detailed information about how ScriptEngine works with YAML scripts can be found in the `ScriptEngine documentation `_. For this documentation it suffices to mention that multiple scripts can be passed to the ScriptEngine ``se`` executable and are processed in order: .. code-block:: shell > se first.yml second.yml third.yml 2023-05-22 11:22:09 INFO [se.cli] Logging configured and started 2023-05-22 11:22:09 INFO [se.task:echo ] Hello from firs... Hello from first script! 2023-05-22 11:22:09 INFO [se.task:echo ] Hello from seco... Hello from second script! 2023-05-22 11:22:09 INFO [se.task:echo <83069dab12>] Hello from thir... Hello from third script! For a brief overview of ``se`` options and arguments, run ``se --help``. .. note:: The actual names of all scripts are just convenient names. There is nothing special about them and they could be changed at will. Scripts could also be split or merged. As far as ScriptEngine is concerned, it will just process, in order, all scripts given as arguments. The basic approach for building the |ece4| components, will be to call the scripts for user settings, the platform settings and the actual compilation commands in one go: .. code-block:: shell se> se user-settings.yml ../platforms/.yml compile-.yml .. note:: It is important to always include the user and platform settings scripts in every compilation! The settings are not magically stored between the ScriptEngine runs (other than in the scripts) and must always be included! The order in which the |ece4| components must be compiled (due to inter-component dependencies) is * OASIS * XIOS * OpenIFS, LPJ-Guess, NEMO, AMIP-Forcing-reader, Runoff-mapper (in any order) .. hint:: There is also a ``compile-components.yml`` script, which compiles selected components in the correct order. The user and platform settings still have to be provided. User settings ~~~~~~~~~~~~~ The ``user-settings.yml`` file is presently very simple and provides some basic configuration for the build process: .. code-block:: yaml+jinja build: # -- Applies to all jobs: 10 clean: True # -- Applies to OpenIFS oifs_exe: # OIFS precision: ['SP-GCM','DP-GCM'] to build both SP and DP, defaults to ['DP-GCM'] # -- Applies to NEMO oasis_active: # NEMO coupled (default: True). Set to `False` for STANDALONE nemo. top_active: # NEMO with TOP (biochemistry/inert tracers). Defaults to `False`. isf_active: # Make option to open ice shelves available. Defaults to `False`. rebuild_nemo: False # Utility to create global files # -- Components list for 'compile-components.yml' # OGCM: [oasis, xios, nemo] # AGCM: [oasis, xios, openifs, amipfr] # AOGCM+Veg (default): [oasis, xios, oifs, nemo, rnfm, amipfr, lpjg] components: where the parameters are: - `experiment.base_dir` : the top directory of the |ece4| code. Here it is set relatively to the current working directory ``se.cli.cwd``. - `experiment.src_dir` : the location of the ``sources`` directory. Here it is set relatively to the ``experiment.base_dir`` - `build.jobs` : number of parallel process to build components, when possible - `build.clean` : if ``true``, the build process will start from scratch (equivalent to ``make clean``) - `build.oifs_exe` : Build OIFS in ``SP-GCM`` (single precision) and ``DP-GCM`` (double precision). If empty or undefined, only ``DP-GCM`` is built. - `build.oasis_active` : switch to build NEMO for standalone (``False``) or coupled-to-OpenIFS (``True``) experiments - `build.top_active` : switch to build NEMO with (``True``) or without (``False``) TOP. TOP handles passive tracers in NEMO, and is required by PISCES and/or for CFCs - `build.isf_active` : switch to build NEMO with (``True``) or without (``False``) ice-shelf functionality. - `build.components` : list of components to compile. Used **ONLY** with ``compile-components.yml``. See :ref:`compiling-components` hereafter Note that NEMO and/or XIOS build directories will differ according to the two ``oasis_active`` and ``top_active`` parameters, allowing for multiple NEMO configurations to be built inside the same source directory. The four possible NEMO configurations are: - `COUPLED` **(default)** – Coupled with OASIS but without passive tracers. - `COUPLED-TOP` – `COUPLED` with passive tracers (e.g. PISCES is activated). - `STANDALONE` – Ocean-only runs, no passive tracers. - `STANDALONE-TOP` – Ocean-only with passive tracers (e.g. water mass age, pisces,...) When ``oasis_active`` and ``top_active`` parameters are empty or undefined, the `COUPLED` configurations of NEMO and XIOS components are built. Note to developers: **xios** is built in its `build_ecearth` dir (for ocean-atmosphere coupled) or `build_no-oasis` dir (for nemo-only). .. _compiling-components: Build with ``compile-components.yml`` ...................................... If you compile several components at once with ``compile-components.yml``, the ``build.components`` list is used. Typically, this list will match your target run configuration, i.e. `model_config.components` (see :ref:`running`). Note that ``build.oasis_active`` is ignored by ``compile-components.yml`` and is instead inferred from the list of components. Examples of possible lists and their corresponding configurations: - **[xios, nemo]** : Ocean-only - **[oasis, xios, oifs, amipfr]** : Atmosphere-only - **[oasis, xios, oifs, nemo, rnfm]** : Coupled Atmosphere-Ocean - **[oasis, xios, oifs, nemo, rnfm, amipfr, lpjg]** : this is the default list (if empty or undefined). All components are compiled, allowing for both coupled Atmosphere-Ocean and Atmosphere-only experiments, but **not** ocean-only ones! .. hint:: The syntax of Python, YAML, Markdown and some other modern languages and formats is based on indentation. Tab characters can not provide consistent indentation, and it has therefore become a major offence to use tab characters in source code! Make sure that your editor replaces tab characters with spaces and that you follow the indentation convention (i.e. how many spaces per level) of the source code you are working with. Again, for details regarding the specific syntax used in the configuration definition, refer to the `ScriptEngine documentation `_, particularly the `Writing Scripts `_ section therein. Platform settings ~~~~~~~~~~~~~~~~~ The settings for different platforms are organised in the ``platforms/`` subdirectory. The platform configuration is naturally a bit heavier than the user settings and comprises things like compiler and library settings, preprocessor macros and other details. Given that there are many aspects and details of platform configuration, the information is structured using a tree-like organisation of configuration parameters. In the configuration scripts for ScriptEngine, this makes use of YAML dictionaries. For example, this is how the compiler-related configuration parameters could be defined for a particular platform: .. code-block:: yaml+jinja base.context: build: lang: fortran: compiler: mpif90 flags: -g -O2 -fdefault-real-8 -fdefault-double-8 -ffree-line-length-none preprocessor: !noparse "{{build.lang.c.preprocessor}}" c: compiler: mpicc flags: -g preprocessor: cpp linker: command: !noparse "{{build.lang.fortran.compiler}}" make: command: gmake As another example, this section to configure some needed libraries could look like: .. code-block:: yaml+jinja base.context: build: # [...] deps: mpi: null lapack: libs: [openblas] grib: base_dir: /software/sse/manual/eccodes/2.8.2/nsc1-gcc-2018a-eb inc_dir: !noparse "{{build.deps.grib.base_dir}}/include" lib_dir: !noparse "{{build.deps.grib.base_dir}}/lib" libs: [eccodes_f90, eccodes, pthread] samples_path: !noparse "{{build.deps.grib.base_dir}}/share/eccodes/samples:\ {{build.deps.grib.base_dir}}/share/eccodes/ifs_samples/grib1_mlgrib2" netcdf: mod_dir: /software/sse/manual/netcdf/4.4.1.1/nsc1-gcc-2018a-eb/include libs: [netcdff, netcdf] hdf5: null Again, the configuration parameters are available for later use in a structured way, such as ``{{build.deps.grib.lib_dir}}``. Some details are worth further explanation in the above example: The syntax ``mpi: null`` and ``hdf5: null`` is used in YAML to denote an empty value. Thus, there is no special MPI or HDF5 configuration on this platform (the configuration is provided by modules in this case). Second, the ``*.libs`` parameters must be defined as YAML lists (with brackets and commas), even if there is only one library. .. hint:: If you want to test user or platform setting scripts, it is possible to run these with ScriptEngine without actual compilation: ``> se user-settings.yml platforms/.yml`` i.e. just omit the compile script(s). Thus, you can make sure the setting scripts are syntactically correct before you attempt the actual compilation. Generally, it will be easiest to start from a known-to-work platform file, copy and adapt to the platform. .. _completing-the-environment: Completing the environment -------------------------- Once the |ece4| components are built, there is one more step before the environment is complete and ready to use for experiments (unless you only intend to run NEMO STANDALONE without OASIS). In order to create grid specific files for the OASIS coupler, |ece4| uses `ready2couple (rdy2cpl) `_, which generates grid description files and regridding weights on-the-fly, according to the grids chosen in the run-time configuration. In order for ready2couple to work, it must be configured to find the OASIS libraries compiled in the step above. Two environment variables must be set/modified for this: - ``OASIS_BUILD_PATH`` must point to the OASIS build directory - the ``LD_LIBRARY_PATH`` variable must include ``${OASIS_BUILD_PATH}/lib`` The ``OASIS_BUILD_PATH`` will usually be something like ``/sources/oasis3-mct-5.2/arch_ecearth``. The correct setting of these variables can be tested by running, with the |ece4| virtual environment activated, the following command: .. code-block:: shell > r2c --help which should produce no error message. .. note:: The OASIS installation process for pyOASIS suggests setting further environment variables. These variables are not needed for ready2couple. .. _machine-specific-builds: Machine-specific builds ----------------------- An alternative way to build the Python environment is as a virtual environment .. code-block:: shell python3.9 -m venv .se source .se/bin/activate pip install -U pip pip install scriptengine pip install scriptengine-tasks-hpc pip install git+https://github.com/ecmwf/ecbundle # After installing OASIS pip install git+https://github.com/uwefladrich/rdy2cpl .. note:: The above method is neither recommended nor supported by the EC-Earth team and should only be used if building the conda environment above is not possible. Puthi-CSC ^^^^^^^^^ For OIFS to run on Puhti (CSC) with GNU, you need edit sources/oifs-48r1/ifs-source/cmake/compile_flags.cmake and comment out the -ffpe-trap=-part. In case you compile with Intel-MPI, do the following before installing ready2cpl: .. code-block:: shell export MPICC=${I_MPI_ROOT}/bin/mpicc export MPILD=”” export MPI4PY_BUILD_CONFIGURE="" pip install --no-cache-dir mpi4py # Tests should be run through srun: srun -n 1 r2c --help srun -n 6 r2c --couple-grid-spec ece_couple_grids.yml namcouple.tmp.yml