Advanced settings

This part of the documentation covers advanced configuration options and performance-related settings for EC-Earth 4 experiments. These settings are often hard-coded or pre-configured per platform, but may require adjustment depending on your specific use case, resolution, precision, and computational setup.

Unlike the simple experiment setup covered in Running simple experiments, this section focuses on tweaking configurations, optimizations and custom model behaviours that experienced users may need to tune for their particular computational environment and requirements.

Specifying these settings is not necessary for most users and they are not present in experiment-config-example.yml by default, but they can be introduced in the experiment configuration file and their values will be used when running the experiment. This is achieved by - when: not model_config.oifs.nproma in the SE config templates that generate the final configuration files for OpenIFS and NEMO.

Resolution-varying parameters in OpenIFS and NEMO

Here we list model parameters in OpenIFS and NEMO which change with grid resolution and give some motivation why. Users can modify these values but should be careful in doing so as it can lead to less realistic climates or numerial issues.

NEMO

changes from eORCA1_ISO to ORCA2

• Increase time step to 5400s for CFL-criterion and faster execution

• Lateral diffusive length scale rn_Ld = 200e+3 for stability

• Lateral viscous length scale rn_Lv = 200.e+3 for stability

changes from eORCA1_ISO to eORCA025

• Decrease time step to 1800s for CFL-criterion

• nn_fct_v=4 for 4th-order vertical advection. nn_fct_v=2 required for _ISO grids.

• Lateral diffusive length rn_Ld = 25e+3 for stability

• Eddy-induced velocities off, ln_ldfeiv=F, since grid can resolve 1st baroclinic Rossby radius in low and middle latitudes.

• Hollingsworth correction, nn_dynkeg=1, to improve mesoscale eddy dynamics.

• Switch Laplacian viscosity to bi-Laplacian, ln_dynldf_blp=T to focus dissipation at smallest scales.

• Lateral viscous velocity and length scales, rn_Uv=0.115 and rn_Lv=25.e+3 for stability.

• Adaptive vertical advection scheme, ln_zad_Aimp=T. Better numerical stability and allows longer time steps.

Note that nn_etau = 1 is default for all configurations but causes unrealistically weak AMOC in ORCA1. The tuning files tuning_v0.3.yml (activated by default) will set nn_etau = 0 and should be used for ORCA1.

OpenIFS

changes from TL255L91 to TL63L31

• Increase time step to 3600s for faster execution

• Radiation calculations every 3 hours, NRADFR = -3

changes from TL255L91 to TCO399L91

• Decrease time step to 900s for stability and ECMWF recommendation <https://confluence.ecmwf.int/display/OIFS/3+Horizontal+Resolution+and+Configurations>

• Turn off filtering of pressure gradients, LGRADSP=F, which is not necessary for TCO grids

changes for TCO1279L137 (high-resolution storyline simulations)

• Very short time step of 450s required for numerical stability at ~9km resolution

• 137 vertical levels for improved vertical resolution

• Typically used for short-term high-resolution storyline simulations rather than long climate runs

• Weather-focused output configuration available via model_config.oifs.output.weather: true

Note

Some model parameters are scaled with grid resolution internally in the model. For example, non-orographic wave drag scales with grid resolution in OpenIFS.

OASIS

It is generally good to have a coupling step short enough to resolve the diurnal cycle, but a shorter coupling time step increases the cost of the model.

coupling steps

• TL255-eORCA1: 5400s. Shortest possible for atm step 1800s and oce step 2700s.

• TL63L31-ORCA2: 10800s. Align with radiation calculation and reduce coupling cost.

• TCO399-eORCA025: 3600s. Align with radiation calculation.

High-resolution storyline simulations

EC-Earth 4 supports ultra-high-resolution simulations for weather storyline studies using the TCO1279L137 grid configuration. This resolution (~9km horizontal, 137 vertical levels) is primarily designed for short-term, high-fidelity simulations that capture detailed weather phenomena and regional climate features.

Use cases

Storyline simulations at TCO1279 resolution are used for detailed studies of weather extremes, regional climate impacts, and atmospheric processes at scales relevant to local decision-making.

Configuration for TCO1279L137

To run a TCO1279L137 simulation, specify the grid in your experiment configuration:

model_config:
  oifs:
    grid: TCO1279L137

The model will automatically configure:

• Time step: 450s (required for numerical stability)

• Vertical levels: 137

• Appropriate OASIS coupling parameters

Initial conditions and nudging

Initial conditions for TCO1279L137 can be downloaded from the OpenIFS Data Hub.

For nudging simulations (Newtonian relaxation to reanalysis), EC-Earth 4 includes tools to generate nudging files from reanalysis data. Nudging helps constrain the model to observed atmospheric states while allowing it to develop small-scale features consistent with the high resolution.

Known limitations

Warning

ICMCL file reading crash (`ECE_CLIMR` issue):

Running with ECE_CLIMR = .TRUE. currently causes crashes during ICMCL file reading in this configuration.

As a workaround (which is reasonable for short nudged runs), ECE_CLIMR must be disabled (disabling AMIP reader OASIS updates in arpifs/ecearth/module/ecearth.F90 and arpifs/climate/updclie_oasis.F90). This can be achieved by commenting out ECE_CLIMR = .TRUE. in the source code prior to compiling:

# Disable ECE_CLIMR in OpenIFS source
sed -i 's/^\(.*ECE_CLIMR = .TRUE.\)/!\1/' sources/oifs-48r1/ifs-source/arpifs/ecearth/module/ecearth.F90
sed -i 's/^\(.*ECE_CLIMR = .TRUE.\)/!\1/' sources/oifs-48r1/ifs-source/surf/module/surfece.F90

This limitation is currently under investigation.

Testing and validation

The TCO1279L137 configuration has been tested and validated on Marenostrum 5 (MN5) using 20–30 nodes with the following setup:

• Test Case: 20 days around the DANA event (October 2024).

• Forcing Data: ICMCL inidata and nudging files are prepared and available for this period.

• Nudging parameters: Vorticity (VO) nudged with a 24-hour relaxation time, applied at vertical levels 70–120 and spectral wavenumbers <= 20.

Weather-focused output

For storyline simulations, you typically need more frequent output and weather-relevant variables. Enable the weather output configuration:

model_config:
  oifs:
    grid: TCO1279L137
    output:
      weather: true

When weather: true is set, the model uses file_def_oifs_weather.xml instead of the default climate-focused output configuration. This provides output optimized for weather analysis and validation.

Computational considerations

TCO1279L137 simulations are computationally expensive:

• Requires substantial computational resources (HPCs with many cores)

• Short time step (450s) means more model steps per simulated period

• Large output files when using weather-focused output

• Typically limited to simulation periods of days to months rather than multi-decadal climate runs

Plan your computational allocation and storage requirements carefully before starting TCO1279 simulations.

Performance-related settings in OpenIFS

NPROMA setting

The NPROMA parameter controls the vertical loop length in the OpenIFS semi-implicit semi-Lagrangian atmospheric model. It represents the number of grid points processed in a single vectorizable loop, which directly affects the computational efficiency and performance of the model.

Optimal value depends on platform, resolution, precision (DP/SP), number of tasks, and OpenMP threads. Currently hardcoded per platform; may need adjustment based on ntasks/nthreads configuration.

For reference, see:

• Comment by Adrian Hill: https://git.smhi.se/ec-earth/ecearth4/-/merge_requests/229#note_206022

• Doubling of optimal NPROMA for SP: https://www.ecmwf.int/en/newsletter/148/meteorology/single-precision-ifs

• suoptproma.F90 for automatic optimization

NPROMA values:

• NPROMA=0: default value is used (Default NPROMA value is 16)

• NPROMA>0: an optimized value is calculated for maximum threads

• NPROMA<0: absolute value is used (guaranteed to use specified value)

OpenIFS output settings

The OpenIFS output configuration controls output frequency, vertical levels, and specific output options:

sample_rate: Controls the sampling frequency for OIFS-XIOS output (FULLPOS)

• If sample_rate > 0: frequency is specified in time steps

• If sample_rate < 0: frequency is specified in hours

• The actual output frequency in seconds is calculated as: sample_rate * dt (for positive values) or -sample_rate * 3600 (for negative values)

pressure_levels: Selects which pressure levels to write

• "pl_1m": write monthly data (default is "plev19")

• "pl_1d": write daily data (default is "false")

• "pl_6hr": write 6-hourly data (default is "false")

Available pressure-level configurations

• "plev3": 3D output on 3 CMIP7 compatible pressure levels (850, 500, 250 hPa)

• "plev7c": 3D output on 7 CMIP7 compatible pressure levels

• "plev8": 3D output on 8 CMIP6 compatible pressure levels

• "plev19": 3D output on 19 CMIP6 & CMIP7 compatible pressure levels

• "plev39": 3D output on 39 pressure levels

• "no": 3D pressure level output disabled

Thus "pl_1d: plev19" will make XIOS write daily output on 19 pressure levels.

Pressure level definitions are specified in axis_def_oifs.xml.j2.

All 3D variables in OpenIFS are defined on 39 pressure levels. Every other pressure-level above is created by XIOS extracting or interpolating from the 39 pressure levels to the target levels.

Other output settings:

• daily_sfc: Enable (true) or disable (false) daily output of surface variables

• cordex_cmip7: Enable (true) or disable (false) boundary conditions output for CORDEX (6-hourly model levels)

• regular_grid: Enable (true) or disable (false) XIOS interpolation from reduced grid to regular grid

• weather: Enable (true) or disable (false, default) weather-focused output configuration. When enabled, uses file_def_oifs_weather.xml instead of file_def_oifs_default.xml. Primarily intended for high-resolution storyline simulations (e.g., TCO1279) that require more frequent output and weather-relevant variables.

• debug_output: Enable (true) or disable (false) XIOS and FULLPOS debug logging

Output configuration templates:

Output settings are managed through templates (not hardcoded in scriptlib files). By default, output-climate.yml is used automatically.

To use a different template:

model_config:
  output_cfg: "{{se.cli.cwd}}/presets/output_cfg/output-weather.yml"

Available templates:

• output-climate.yml (default): Standard monthly output for long climate simulations (serves as the reference template for custom setups)

• output-weather.yml: High-frequency output for weather storyline simulations

NEMO output settings

The NEMO output configuration controls how ocean model data is written. The following parameters control mesh output, initial state output, and tracer diagnostics:

meshmask: Activate writing of mesh mask files

istate: Activate output of initial state

cordex: Activate output for CORDEX boundary conditions

trdtrc: Activate tracer trends diagnostic output

• Requires the key_trdtrc compilation key to be enabled

• tracers: List of tracer names (e.g., [POC, PHY, ZOO, DOC, PHY2, ZOO2, GOC]) to include in diagnostic output

• trends: List of trend names (e.g., [XAD, YAD, ZAD, LDF, ZDF, SMS, TOT]) to include in diagnostic output

Modifications should be introduced in the output configuration templates (e.g., scripts/runtime/presets/output_cfg/output-climate.yml) directly.

XIOS settings

The output from OpenIFS and NEMO can be modified and controlled in the config-xios.yml file or the templates/xios/*xml.j2 files.

Inspect XIOS performance

XIOS will write a performance report to the experiment log file after a successful job. For example

report :  Performance report : Ratio : 0.00290895 %
report :  Performance report : This ratio must be close to zero. Otherwise it may be useful to increase buffer size or numbers of server

As the message suggests, the Ratio should be less than a few %. If not, consider tuning XIOS performance.

Improving XIOS performance

In scripts/runtime/scriptlib/config-xios.yml:

• model_config.xios.compression_level: Compression level (default 0) of output. 1 can compress output by 20-40% without impacting performance.

• model_config.xios.xios3_finetuning: Enable (true) or disable (false) defining writers and gatherers in XIOS3 to improve performace. Requires 4 XIOS tasks for OpenIFS and 8 for NEMO.

Both options are off by default. Note that the xios3_finetuning option has not been evaluated. More information about “writers” and “gatherers” in XIOS3 is found here<https://sites.nemo-ocean.io/user-guide/xios3demo.html>.

• sync_freq at the top of the file_def files, e.g. file_def_nemo-oce.xml.j2 should be 1 month. Higher frequency, e.g. 1day, will slow down XIOS

• buffer_size_factor in iodef.xml.j2 should be larger than 1 (default is 4) to allow XIOS large buffers

• optimal_buffer_size in iodef.xml.j2 can be performance (more memory for more speed) or memory (minimise memory use but may take performance hit). Default is performance, but memory may be necessary for high-resolution experiments.

Tips and tricks

XIOS can output a subset of vertical levels. It requires defining a new axis and its relation to the original axis. For example, assume your output is written on 39 pressure levels. The pressure levels in is then axis_def_oifs.xml.j2

<axis id="pressure_levels" long_name="vertical pressure levels" n_glo="39" value="(0,38)[100000.0 92500.0 85000.0 70000.0 60000.0 50000.0 40000.0 30000.0 25000.0 20000.0 17000.0 15000.0 13000.0 11500.0 10000.0 9000.0 8000.0 7000.0 5000.0 3000.0 2000.0 1500.0 1000.0 700.0 500.0 300.0 200.0 150.0 100.0 70.0 50.0 40.0 30.0 20.0 15.0 10.0 7.0 5.0 3.0]"/>

Then define a new vertical axis in axis_def_oifs.xml.j2

<axis id="pressure_levels_plev8" long_name="plev8 pressure levels" axis_ref="pressure_levels" n_glo="8" value="(0,7)[100000.0 85000.0 70000.0 50000.0 25000.0 10000.0 5000.0 1000.0]">
  <interpolate_axis type="polynomial" order="1" />
</axis>

This will define a new vertical pressure axis, pressure_levels_plev8, by linear interpolation from the 39-layer original axis. Note how the new axis is based on axis_ref="pressure_levels", which instructs XIOS which axis to interpolate from. Next, define a grid in grid_def_oifs.xml.j2 using the new axis

<grid_group id="pl_grids">
  <grid id="reduced_pl" description="3D model grid with pressure levels">
    <domain domain_ref="reduced_gaussian"/>
    <axis axis_ref="pressure_levels"/>
  </grid>
  <grid id="reduced_pl_plev8">
    <domain domain_ref="reduced_gaussian"/>
    <axis axis_ref="pressure_levels_plev8"/>
  </grid>
</grid_group>

Next, define output on the new grid in file_def_oifs_default.xml.j2

   <file enabled="true" output_freq="6h" name_suffix="_plev8_6h" >
     <field_group operation="average" grid_ref="reduced_pl_plev8" >
       <field field_ref="air_temperature__ta"/>
     </field_group>
   </file>

which will write 6-hourly air temperature on 8 pressure levels.

Note

OpenIFS uses terrain-following model levels where the pressure varies in both space and time. The interpolation from the hybrid sigma-pressure levels to pressure levels is done by FullPos inside OpenIFS and the resulting pressure-level fields are then sent to XIOS. It is not possible to interpolate from model levels to pressure levels in XIOS.

Running atmosphere-only with custom SST and sea-ice data

The default atmosphere-only experiment in EC-Earth is to use the SST and sea-ice data provided by input4MIP which is monthly and on a regular lat-lon grid of size 180x360. It is possible to use SST and sea-ice data from other sources but it requires some tweaks. Here follows an example on how to use OSTIA data which is daily and on a 0.05-degree grid.

First, we need to adapt the data for use.

# - Take daily data for 2018
# - interpolate to 0.2deg grid
# - change K->degC
# - fill missing values
cdo -L -fillmiss
       -expr,"tosbcs=tosbcs-273.15"
       -remapbil,r1800x900
       -chname,analysed_sst,tosbcs
       -select,name=analysed_sst,startdate=2018-01-01T00:00:00,enddate=2019-01-01T00:00:00
       OSTIA/METOFFICE-GLO-SST-L4-REP-OBS-SST/????/??/ostia_rep_global_????-??-??.nc sst_tmp.nc

# time axis must be floats since time is 12:00:00 and we need units of days, i.e. 0.5 days
ncap2 -O -s 'time=float(time)' sst_tmp.nc sst_tmp_f.nc

# set ref time to days since start
cdo -L -setreftime,1980-01-01,00:00:00,1day sst_tmp_f.nc sst_OSTIA_1d_1800x900_filled_2018.nc

# - Take daily data for 2018
# - interpolate to 0.2deg grid
# - set missing values to 0
cdo -L -fillmiss
       -expr,"siconcbcs=siconcbcs*100"
       -remapbil,r1800x900
       -chname,sea_ice_fraction,siconcbcs
       -select,name=sea_ice_fraction,startdate=2018-01-01T00:00:00,enddate=2019-01-01T00:00:00
       OSTIA/METOFFICE-GLO-SST-L4-REP-OBS-SST/????/??/ostia_rep_global_????-??-??.nc sic_tmp.nc

ncap2 -O -s 'time=float(time)' sic_tmp.nc sic_tmp_f.nc

cdo -L -setreftime,1980-01-01,00:00:00,1day sic_tmp_f.nc sic_OSTIA_1d_1800x900_filled_1980-2023.nc

In the example above we have daily data from OSTIA which we merge into a single file, then convert units for data and time. The data is now on a 0.2-degree grid (900x1800). Note that we have changed the variable names to tosbcs (SST) and siconcbcs (sea ice) and set units to Celsius and percent which is what the AMIP forcing reader expects.

We now need to modify our experiment configuration to make EC-Earth accept the files

- base.context:
    experiment:
      forcing:
        amipfr:
          dir: /nobackup/accmls1/proj/scanheat/satellite_data/forcing_for_ece4/
          sic: sic_OSTIA_1d_1800x900_filled_1980-2023.nc
          sst: sst_OSTIA_1d_1800x900_filled_1980-2023.nc
          yref_min: 2018 # first year of data set
          yref_max: 2018 # last year of data set

- base.context:
    model_config:
      amipfr:
        grid: r900x1800

      oasis:
        copy_weights: False

The above additions to your experiment configuration file will tell EC-Earth to use your new files, on the new grid, and to generate new remapping weights for OASIS.

Note

Your grid must be defined in templates/oasis/ece_couple_grids.yml. Only r180x360 and r900x1800 are defined so far, but you can add more.