.. _running_advanced: Advanced settings ================= This part of the documentation covers advanced configuration options and performance-related settings for |ece4| experiments. These settings are often hard-coded or pre-configured per platform, but may require adjustment depending on your specific use case, resolution, precision, and computational setup. Unlike the simple experiment setup covered in :ref:`running`, this section focuses on tweaking configurations, optimizations and custom model behaviours that experienced users may need to tune for their particular computational environment and requirements. Specifying these settings is not necessary for most users and they are not present in ``experiment-config-example.yml`` by default, but they can be introduced in the experiment configuration file and their values will be used when running the experiment. This is achieved by ``- when: not model_config.oifs.nproma`` in the SE config templates that generate the final configuration files for OpenIFS and NEMO. Resolution-varying parameters in OpenIFS and NEMO ------------------------------------------------- Here we list model parameters in OpenIFS and NEMO which change with grid resolution and give some motivation why. Users can modify these values but should be careful in doing so as it can lead to less realistic climates or numerial issues. NEMO ~~~~ **changes from eORCA1_ISO to ORCA2** - Increase time step to 5400s for CFL-criterion and faster execution - Lateral diffusive length scale ``rn_Ld = 200e+3`` for stability - Lateral viscous length scale ``rn_Lv = 200.e+3`` for stability **changes from eORCA1_ISO to eORCA025** - Decrease time step to 1800s for CFL-criterion - ``nn_fct_v=4`` for 4th-order vertical advection. ``nn_fct_v=2`` required for ``_ISO`` grids. - Lateral diffusive length ``rn_Ld = 25e+3`` for stability - Eddy-induced velocities off, ``ln_ldfeiv=F``, since grid can resolve 1st baroclinic Rossby radius in low and middle latitudes. - Hollingsworth correction, ``nn_dynkeg=1``, to improve mesoscale eddy dynamics. - Switch Laplacian viscosity to bi-Laplacian, ``ln_dynldf_blp=T`` to focus dissipation at smallest scales. - Lateral viscous velocity and length scales, ``rn_Uv=0.115`` and ``rn_Lv=25.e+3`` for stability. - Adaptive vertical advection scheme, ``ln_zad_Aimp=T``. Better numerical stability and allows longer time steps. Note that ``nn_etau = 1`` is default for all configurations but causes unrealistically weak AMOC in ORCA1. The tuning files ``tuning_v0.3.yml`` (activated by default) will set ``nn_etau = 0`` and should be used for ORCA1. OpenIFS ~~~~~~~ **changes from TL255L91 to TL63L31** - Increase time step to 3600s for faster execution - Radiation calculations every 3 hours, ``NRADFR = -3`` **changes from TL255L91 to TCO399L91** - Decrease time step to 900s for stability and `ECMWF recommendation ` - Turn off filtering of pressure gradients, ``LGRADSP=F``, which is not necessary for TCO grids .. note:: Some model parameters are scaled with grid resolution internally in the model. For example, non-orographic wave drag scales with grid resolution in OpenIFS. OASIS ~~~~~ It is generally good to have a coupling step short enough to resolve the diurnal cycle, but a shorter coupling time step increases the cost of the model. **coupling steps** - TL255-eORCA1: 5400s. Shortest possible for atm step 1800s and oce step 2700s. - TL63L31-ORCA2: 10800s. Align with radiation calculation and reduce coupling cost. - TCO399-eORCA025: 3600s. Align with radiation calculation. Performance-related settings in OpenIFS --------------------------------------- NPROMA setting ~~~~~~~~~~~~~~ The NPROMA parameter controls the vertical loop length in the OpenIFS semi-implicit semi-Lagrangian atmospheric model. It represents the number of grid points processed in a single vectorizable loop, which directly affects the computational efficiency and performance of the model. Optimal value depends on platform, resolution, precision (DP/SP), number of tasks, and OpenMP threads. Currently hardcoded per platform; may need adjustment based on ntasks/nthreads configuration. For reference, see: - Comment by Adrian Hill: https://git.smhi.se/ec-earth/ecearth4/-/merge_requests/229#note_206022 - Doubling of optimal NPROMA for SP: https://www.ecmwf.int/en/newsletter/148/meteorology/single-precision-ifs - ``suoptproma.F90`` for automatic optimization NPROMA values: - ``NPROMA=0``: default value is used (Default NPROMA value is 16) - ``NPROMA>0``: an optimized value is calculated for maximum threads - ``NPROMA<0``: absolute value is used (guaranteed to use specified value) OpenIFS output settings ~~~~~~~~~~~~~~~~~~~~~~~ The OpenIFS output configuration controls output frequency, vertical levels, and specific output options: **sample_rate**: Controls the sampling frequency for OIFS-XIOS output (FULLPOS) - If ``sample_rate > 0``: frequency is specified in time steps - If ``sample_rate < 0``: frequency is specified in hours - The actual output frequency in seconds is calculated as: ``sample_rate * dt`` (for positive values) or ``-sample_rate * 3600`` (for negative values) **pressure_levels**: Selects which pressure levels to write - ``"pl_1m"``: write monthly data (default is ``"plev19"``) - ``"pl_1d"``: write daily data (default is ``"false"``) - ``"pl_6hr"``: write 6-hourly data (default is ``"false"``) Available pressure-level configurations - ``"plev3"``: 3D output on 3 CMIP7 compatible pressure levels (850, 500, 250 hPa) - ``"plev7c"``: 3D output on 7 CMIP7 compatible pressure levels - ``"plev8"``: 3D output on 8 CMIP6 compatible pressure levels - ``"plev19"``: 3D output on 19 CMIP6 & CMIP7 compatible pressure levels - ``"plev39"``: 3D output on 39 pressure levels - ``"no"``: 3D pressure level output disabled Thus ``"pl_1d: plev19"`` will make XIOS write daily output on 19 pressure levels. Pressure level definitions are specified in ``axis_def_oifs.xml.j2``. All 3D variables in OpenIFS are defined on 39 pressure levels. Every other pressure-level above is created by XIOS extracting or interpolating from the 39 pressure levels to the target levels. **Other output settings**: - ``daily_sfc``: Enable (``yes``) or disable (``no``) daily output of surface variables - ``pressure_levels_zoom``: Enable (``true``) or disable (``no``) 6-hourly output on 850, 500 and 300 hPa - ``cordex_cmip7``: Enable (``yes``) or disable (``no``) boundary conditions output for CORDEX (6-hourly model levels) - ``regular_grid``: Enable (``yes``) or disable (``no``) XIOS interpolation from reduced grid to regular grid - ``debug_output``: Enable (``true``) or disable (``false``) XIOS and FULLPOS debug logging Modifications should be introduced in ``scripts/runtime/scriptlib/config-oifs.yml`` directly. NEMO output settings ~~~~~~~~~~~~~~~~~~~~ The NEMO output configuration controls how ocean model data is written. The following parameters control mesh output, initial state output, and tracer diagnostics: **meshmask**: Activate writing of mesh mask files **istate**: Activate output of initial state **cordex**: Activate output for CORDEX boundary conditions **trdtrc**: Activate tracer trends diagnostic output - Requires the ``key_trdtrc`` compilation key to be enabled - **tracers**: List of tracer names (e.g., ``[POC, PHY, ZOO, DOC, PHY2, ZOO2, GOC]``) to include in diagnostic output - **trends**: List of trend names (e.g., ``[XAD, YAD, ZAD, LDF, ZDF, SMS, TOT]``) to include in diagnostic output Modifications should be introduced in ``scripts/runtime/scriptlib/config-nemo.yml`` directly. XIOS settings ~~~~~~~~~~~~~ The output from OpenIFS and NEMO can be modified and controlled in the ``config-xios.yml`` file or the ``templates/xios/*xml.j2`` files. **Inspect XIOS performance** XIOS will write a performance report to the experiment log file after a successful job. For example .. code-block:: shell report : Performance report : Ratio : 0.00290895 % report : Performance report : This ratio must be close to zero. Otherwise it may be useful to increase buffer size or numbers of server As the message suggests, the ``Ratio`` should be less than a few %. If not, consider tuning XIOS performance. **Improving XIOS performance** In ``scripts/runtime/scriptlib/config-xios.yml``: - ``model_config.xios.compression_level``: Compression level (default ``0``) of output. ``1`` can compress output by 20-40% without impacting performance. - ``model_config.xios.xios3_finetuning``: Enable (``true``) or disable (``false``) defining writers and gatherers in XIOS3 to improve performace. Requires 4 XIOS tasks for OpenIFS and 8 for NEMO. Both options are off by default. Note that the ``xios3_finetuning`` option has not been evaluated. More information about "writers" and "gatherers" in XIOS3 is found `here`. - ``sync_freq`` at the top of the ``file_def`` files, e.g. ``file_def_nemo-oce.xml.j2`` should be 1 month. Higher frequency, e.g. 1day, will slow down XIOS - ``buffer_size_factor`` in ``iodef.xml.j2`` should be larger than ``1`` (default is ``4``) to allow XIOS large buffers - ``optimal_buffer_size`` in ``iodef.xml.j2`` can be ``performance`` (more memory for more speed) or ``memory`` (minimise memory use but may take performance hit). Default is ``performance``, but ``memory`` may be necessary for high-resolution experiments. **Tips and tricks** XIOS can output a subset of vertical levels. It requires defining a new axis and its relation to the original axis. For example, assume your output is written on 39 pressure levels. The pressure levels in is then ``axis_def_oifs.xml.j2`` .. code-block:: xml Then define a new vertical axis in ``axis_def_oifs.xml.j2`` .. code-block:: xml This will define a new vertical pressure axis, ``pressure_levels_plev8``, by linear interpolation from the 39-layer original axis. Note how the new axis is based on ``axis_ref="pressure_levels"``, which instructs XIOS which axis to interpolate from. Next, define a grid in ``grid_def_oifs.xml.j2`` using the new axis .. code-block:: xml Next, define output on the new grid in ``file_def_oifs_default.xml.j2`` .. code-block:: xml which will write 6-hourly air temperature on 8 pressure levels. .. note:: OpenIFS uses terrain-following model levels where the pressure varies in both space and time. The interpolation from the hybrid sigma-pressure levels to pressure levels is done by FullPos inside OpenIFS and the resulting pressure-level fields are then sent to XIOS. It is not possible to interpolate from model levels to pressure levels in XIOS. Running atmosphere-only with custom SST and sea-ice data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The default atmosphere-only experiment in EC-Earth is to use the SST and sea-ice data provided by input4MIP which is monthly and on a regular lat-lon grid of size 180x360. It is possible to use SST and sea-ice data from other sources but it requires some tweaks. Here follows an example on how to use OSTIA data which is daily and on a 0.05-degree grid. First, we need to adapt the data for use. .. code-block:: shell # - Take daily data for 2018 # - interpolate to 0.2deg grid # - change K->degC # - fill missing values cdo -L -fillmiss -expr,"tosbcs=tosbcs-273.15" -remapbil,r1800x900 -chname,analysed_sst,tosbcs -select,name=analysed_sst,startdate=2018-01-01T00:00:00,enddate=2019-01-01T00:00:00 OSTIA/METOFFICE-GLO-SST-L4-REP-OBS-SST/????/??/ostia_rep_global_????-??-??.nc sst_tmp.nc # time axis must be floats since time is 12:00:00 and we need units of days, i.e. 0.5 days ncap2 -O -s 'time=float(time)' sst_tmp.nc sst_tmp_f.nc # set ref time to days since start cdo -L -setreftime,1980-01-01,00:00:00,1day sst_tmp_f.nc sst_OSTIA_1d_1800x900_filled_2018.nc # - Take daily data for 2018 # - interpolate to 0.2deg grid # - set missing values to 0 cdo -L -fillmiss -expr,"siconcbcs=siconcbcs*100" -remapbil,r1800x900 -chname,sea_ice_fraction,siconcbcs -select,name=sea_ice_fraction,startdate=2018-01-01T00:00:00,enddate=2019-01-01T00:00:00 OSTIA/METOFFICE-GLO-SST-L4-REP-OBS-SST/????/??/ostia_rep_global_????-??-??.nc sic_tmp.nc ncap2 -O -s 'time=float(time)' sic_tmp.nc sic_tmp_f.nc cdo -L -setreftime,1980-01-01,00:00:00,1day sic_tmp_f.nc sic_OSTIA_1d_1800x900_filled_1980-2023.nc In the example above we have daily data from OSTIA which we merge into a single file, then convert units for data and time. The data is now on a 0.2-degree grid (900x1800). Note that we have changed the variable names to ``tosbcs`` (SST) and ``siconcbcs`` (sea ice) and set units to Celsius and percent which is what the AMIP forcing reader expects. We now need to modify our experiment configuration to make EC-Earth accept the files .. code-block:: yaml - base.context: experiment: forcing: amipfr: dir: /nobackup/accmls1/proj/scanheat/satellite_data/forcing_for_ece4/ sic: sic_OSTIA_1d_1800x900_filled_1980-2023.nc sst: sst_OSTIA_1d_1800x900_filled_1980-2023.nc yref_min: 2018 # first year of data set yref_max: 2018 # last year of data set - base.context: model_config: amipfr: grid: r900x1800 oasis: copy_weights: False The above additions to your experiment configuration file will tell EC-Earth to use your new files, on the new grid, and to generate new remapping weights for OASIS. .. note:: Your grid must be defined in ``templates/oasis/ece_couple_grids.yml``. Only ``r180x360`` and ``r900x1800`` are defined so far, but you can add more.