NPROMA setting
The NPROMA parameter controls the vertical loop length in the OpenIFS
semi-implicit semi-Lagrangian atmospheric model. It represents the number of
grid points processed in a single vectorizable loop, which directly affects
the computational efficiency and performance of the model.
Optimal value depends on platform, resolution, precision (DP/SP), number of tasks, and OpenMP threads.
Currently hardcoded per platform; may need adjustment based on ntasks/nthreads configuration.
For reference, see:
NPROMA values:
NPROMA=0: default value is used (Default NPROMA value is 16)
NPROMA>0: an optimized value is calculated for maximum threads
NPROMA<0: absolute value is used (guaranteed to use specified value)
OpenIFS output settings
The OpenIFS output configuration controls output frequency, vertical levels, and specific output options:
sample_rate: Controls the sampling frequency for OIFS-XIOS output (FULLPOS)
If sample_rate > 0: frequency is specified in time steps
If sample_rate < 0: frequency is specified in hours
The actual output frequency in seconds is calculated as: sample_rate * dt (for positive values) or -sample_rate * 3600 (for negative values)
pressure_levels: Selects which pressure levels to write
"pl_1m": write monthly data (default is "plev19")
"pl_1d": write daily data (default is "false")
"pl_6hr": write 6-hourly data (default is "false")
Available pressure-level configurations
"plev3": 3D output on 3 CMIP7 compatible pressure levels (850, 500, 250 hPa)
"plev7c": 3D output on 7 CMIP7 compatible pressure levels
"plev8": 3D output on 8 CMIP6 compatible pressure levels
"plev19": 3D output on 19 CMIP6 & CMIP7 compatible pressure levels
"plev39": 3D output on 39 pressure levels
"no": 3D pressure level output disabled
Thus "pl_1d: plev19" will make XIOS write daily output on 19 pressure levels.
Pressure level definitions are specified in axis_def_oifs.xml.j2.
All 3D variables in OpenIFS are defined on 39 pressure levels.
Every other pressure-level above is created by XIOS extracting or interpolating from the 39 pressure levels to the target levels.
Other output settings:
daily_sfc: Enable (yes) or disable (no) daily output of surface variables
pressure_levels_zoom: Enable (true) or disable (no) 6-hourly output on 850, 500 and 300 hPa
cordex_cmip7: Enable (yes) or disable (no) boundary conditions output for CORDEX (6-hourly model levels)
regular_grid: Enable (yes) or disable (no) XIOS interpolation from reduced grid to regular grid
debug_output: Enable (true) or disable (false) XIOS and FULLPOS debug logging
Modifications should be introduced in scripts/runtime/scriptlib/config-oifs.yml directly.
NEMO output settings
The NEMO output configuration controls how ocean model data is written. The following parameters control mesh output, initial state output, and tracer diagnostics:
meshmask: Activate writing of mesh mask files
istate: Activate output of initial state
cordex: Activate output for CORDEX boundary conditions
trdtrc: Activate tracer trends diagnostic output
Requires the key_trdtrc compilation key to be enabled
tracers: List of tracer names (e.g., [POC, PHY, ZOO, DOC, PHY2, ZOO2, GOC]) to include in diagnostic output
trends: List of trend names (e.g., [XAD, YAD, ZAD, LDF, ZDF, SMS, TOT]) to include in diagnostic output
Modifications should be introduced in scripts/runtime/scriptlib/config-nemo.yml directly.
XIOS settings
The output from OpenIFS and NEMO can be modified and controlled in the config-xios.yml file or the templates/xios/*xml.j2 files.
Inspect XIOS performance
XIOS will write a performance report to the experiment log file after a successful job.
For example
report : Performance report : Ratio : 0.00290895 %
report : Performance report : This ratio must be close to zero. Otherwise it may be useful to increase buffer size or numbers of server
As the message suggests, the Ratio should be less than a few %. If not, consider tuning XIOS performance.
Improving XIOS performance
In scripts/runtime/scriptlib/config-xios.yml:
model_config.xios.compression_level: Compression level (default 0) of output. 1 can compress output by 20-40% without impacting performance.
model_config.xios.xios3_finetuning: Enable (true) or disable (false) defining writers and gatherers in XIOS3 to improve performace. Requires 4 XIOS tasks for OpenIFS and 8 for NEMO.
Both options are off by default. Note that the xios3_finetuning option has not been evaluated.
More information about “writers” and “gatherers” in XIOS3 is found here<https://sites.nemo-ocean.io/user-guide/xios3demo.html>.
sync_freq at the top of the file_def files, e.g. file_def_nemo-oce.xml.j2 should be 1 month. Higher frequency, e.g. 1day, will slow down XIOS
buffer_size_factor in iodef.xml.j2 should be larger than 1 (default is 4) to allow XIOS large buffers
optimal_buffer_size in iodef.xml.j2 can be performance (more memory for more speed) or memory (minimise memory use but may take performance hit). Default is performance, but memory may be necessary for high-resolution experiments.
Tips and tricks
XIOS can output a subset of vertical levels. It requires defining a new axis and its relation to the original axis.
For example, assume your output is written on 39 pressure levels. The pressure levels in is then axis_def_oifs.xml.j2
<axis id="pressure_levels" long_name="vertical pressure levels" n_glo="39" value="(0,38)[100000.0 92500.0 85000.0 70000.0 60000.0 50000.0 40000.0 30000.0 25000.0 20000.0 17000.0 15000.0 13000.0 11500.0 10000.0 9000.0 8000.0 7000.0 5000.0 3000.0 2000.0 1500.0 1000.0 700.0 500.0 300.0 200.0 150.0 100.0 70.0 50.0 40.0 30.0 20.0 15.0 10.0 7.0 5.0 3.0]"/>
Then define a new vertical axis in axis_def_oifs.xml.j2
<axis id="pressure_levels_plev8" long_name="plev8 pressure levels" axis_ref="pressure_levels" n_glo="8" value="(0,7)[100000.0 85000.0 70000.0 50000.0 25000.0 10000.0 5000.0 1000.0]">
<interpolate_axis type="polynomial" order="1" />
</axis>
This will define a new vertical pressure axis, pressure_levels_plev8, by linear interpolation from the 39-layer original axis.
Note how the new axis is based on axis_ref="pressure_levels", which instructs XIOS which axis to interpolate from.
Next, define a grid in grid_def_oifs.xml.j2 using the new axis
<grid_group id="pl_grids">
<grid id="reduced_pl" description="3D model grid with pressure levels">
<domain domain_ref="reduced_gaussian"/>
<axis axis_ref="pressure_levels"/>
</grid>
<grid id="reduced_pl_plev8">
<domain domain_ref="reduced_gaussian"/>
<axis axis_ref="pressure_levels_plev8"/>
</grid>
</grid_group>
Next, define output on the new grid in file_def_oifs_default.xml.j2
<file enabled="true" output_freq="6h" name_suffix="_plev8_6h" >
<field_group operation="average" grid_ref="reduced_pl_plev8" >
<field field_ref="air_temperature__ta"/>
</field_group>
</file>
which will write 6-hourly air temperature on 8 pressure levels.
Note
OpenIFS uses terrain-following model levels where the pressure varies in both space and time.
The interpolation from the hybrid sigma-pressure levels to pressure levels is done by FullPos
inside OpenIFS and the resulting pressure-level fields are then sent to XIOS.
It is not possible to interpolate from model levels to pressure levels in XIOS.
Running atmosphere-only with custom SST and sea-ice data
The default atmosphere-only experiment in EC-Earth is to use the SST and sea-ice data provided by input4MIP which is monthly and on a regular lat-lon grid of size 180x360.
It is possible to use SST and sea-ice data from other sources but it requires some tweaks.
Here follows an example on how to use OSTIA data which is daily and on a 0.05-degree grid.
First, we need to adapt the data for use.
# - Take daily data for 2018
# - interpolate to 0.2deg grid
# - change K->degC
# - fill missing values
cdo -L -fillmiss
-expr,"tosbcs=tosbcs-273.15"
-remapbil,r1800x900
-chname,analysed_sst,tosbcs
-select,name=analysed_sst,startdate=2018-01-01T00:00:00,enddate=2019-01-01T00:00:00
OSTIA/METOFFICE-GLO-SST-L4-REP-OBS-SST/????/??/ostia_rep_global_????-??-??.nc sst_tmp.nc
# time axis must be floats since time is 12:00:00 and we need units of days, i.e. 0.5 days
ncap2 -O -s 'time=float(time)' sst_tmp.nc sst_tmp_f.nc
# set ref time to days since start
cdo -L -setreftime,1980-01-01,00:00:00,1day sst_tmp_f.nc sst_OSTIA_1d_1800x900_filled_2018.nc
# - Take daily data for 2018
# - interpolate to 0.2deg grid
# - set missing values to 0
cdo -L -fillmiss
-expr,"siconcbcs=siconcbcs*100"
-remapbil,r1800x900
-chname,sea_ice_fraction,siconcbcs
-select,name=sea_ice_fraction,startdate=2018-01-01T00:00:00,enddate=2019-01-01T00:00:00
OSTIA/METOFFICE-GLO-SST-L4-REP-OBS-SST/????/??/ostia_rep_global_????-??-??.nc sic_tmp.nc
ncap2 -O -s 'time=float(time)' sic_tmp.nc sic_tmp_f.nc
cdo -L -setreftime,1980-01-01,00:00:00,1day sic_tmp_f.nc sic_OSTIA_1d_1800x900_filled_1980-2023.nc
In the example above we have daily data from OSTIA which we merge into a single file, then convert units for data and time.
The data is now on a 0.2-degree grid (900x1800).
Note that we have changed the variable names to tosbcs (SST) and siconcbcs (sea ice) and set units to Celsius and percent which is what the AMIP forcing reader expects.
We now need to modify our experiment configuration to make EC-Earth accept the files
- base.context:
experiment:
forcing:
amipfr:
dir: /nobackup/accmls1/proj/scanheat/satellite_data/forcing_for_ece4/
sic: sic_OSTIA_1d_1800x900_filled_1980-2023.nc
sst: sst_OSTIA_1d_1800x900_filled_1980-2023.nc
yref_min: 2018 # first year of data set
yref_max: 2018 # last year of data set
- base.context:
model_config:
amipfr:
grid: r900x1800
oasis:
copy_weights: False
The above additions to your experiment configuration file will tell EC-Earth to use your new files, on the new grid, and to generate new remapping weights for OASIS.
Note
Your grid must be defined in templates/oasis/ece_couple_grids.yml.
Only r180x360 and r900x1800 are defined so far, but you can add more.