Running hofx 3D application in JEDI

Running hofx 3D application in JEDI#

This tutorial assumes that you have set up your work environment (which can be done by loading Spack-Stack modules and that you have built jedi-bundle.

DISCOVER users can use the jedi_bundle tool to build JEDI on DISCOVER with the latest Spack Stack modules by following instructions here.

prerequisite#

  • jedi-bundle is already built in your JEDI_BUILD.

  • Background files

  • observation files

  • Input geometry files

Introducton:#

An H(x) (hofx) application computes model equivalent values at the locations and times of observations using observation operators. Depending on the observation type, observation operator can either do a simple interpolation of model output to the observation location or a more complicated operation when the observation values are not avaiable in the model outptus.

For example, Aerosol Optical Depth (AOD) is a prognostic quantity that is not typically stored in model outputs. To calculate hofx values, the AOD observation operator calculates AOD from the model state (x) using the CRTM observation operator. In JEDI, observation operators are located in the Unified Forward Operator (UFO) code repository.

Another example is the column retreval operator that can be used for profile and column satellite retrievals. This operator has been used for NO2 tropospheric and total and CO total columns measurements by TEMPO and TROPOMI.

In this section, we introduce two approaches for computing H(x) in JEDI: Hofx3D and Hofx4D. Each approach makes different simplifying assumptions about the observation time to reduce computational cost.

HofX3D#

The HofX3D approach uses the model state (blue circle) valid at the middle of the assimilation window and assumes that all observations (red circles) within the assimialtion window are also valid at the same time. The model values at the observation locations are then spatially interpolated. Because computing the model state at each individual observation time would be computationally costly, we assume that all the observations (within the window) were measured at the same time as our model state, which is in the middle of the window.

In FV3-JEDI, the executable for the HofX3D application is fv3jedi_hofx_nomodel.x

YAML structure (HofX3D):#

To run fv3jedi_hofx_nomodel.x, JEDI requires in input file (in YAML format) which specifies the assimilation window, input geometry and state information, input observation information, and the observation operator. Below is an example to run the Column Retrieval observation operation with TEMPO observation and a low resolution GEOS-CF background.

# Beginning and length of assimilation window
time window:
  begin: '2023-08-05T15:00:00Z'
  length: PT6H

# Geometry of the state or background
geometry:
  fms initialization:
    namelist filename: inputs/geometry_input/fmsmpp.nml
  akbk: inputs/geometry_input/akbk72.nc4
  npx: 13
  npy: 13
  npz: 72

# Model state valid in the middle of the assimilation window
state:
  datetime: '2023-08-05T18:00:00Z'
  filetype: cube sphere history
  datapath: inputs/geos_cf_c12
  filename: CF2.geoscf_jedi.c12.20230805T180000Z.nc4
  state variables:
  - air_pressure_thickness
  - volume_mixing_ratio_of_no2
  - volume_mixing_ratio_of_no
  - volume_mixing_ratio_of_o3
  - air_pressure_at_surface
  field io names:
    air_pressure_thickness: DELP
    volume_mixing_ratio_of_no: 'NO'
    volume_mixing_ratio_of_no2: NO2
    volume_mixing_ratio_of_o3: O3
    air_pressure_at_surface: PS
  max allowable geometry difference: 0.1
                    
# Observations measured within the assimilation window
# But are assumed to be valid in the middle of the assimilation window.
observations:
  observers:
  - obs space:
      name: tempo_no2_tropo
      obsdatain:
        engine:
          obsfile: inputs/obs/tempo_no2_tropo_20230805T150000Z.nc
          type: H5File
      obsdataout:
        engine:
          allow overwrite: true
          obsfile: output/fb.hofx3d.tempo_no2_tropo.20230805T150000Z.nc
          type: H5File
      observed variables:
      - nitrogendioxideColumn
      simulated variables:
      - nitrogendioxideColumn
    obs operator:
      name: ColumnRetrieval
      isApriori: false
      isAveragingKernel: true
      nlayers_retrieval: 72
      stretchVertices: topbottom
      tracer variables:
      - volume_mixing_ratio_of_no2

A few points about this experiment’s YAML file:

  • Note that the assimilation window begins at 2023-08-05T15:00:00Z and has a length of 6h (PT6H)

  • The model state is valid at 2023-08-05T18:00:00Z which is the middle of the window.

  • If you examine the observation file (inputs/obs/tempo_no2_tropo_20230805T150000Z.nc), you can see that it includes measurements from 15Z to 21Z.

  • The observation operator is ColumnRetrieval which calculates tropospheric column NO2 using averaging kernals.

  • The output is saved at output/fb.tempo_no2_tropo.20230805T150000Z.nc. This file is a nested NetCDF in the IODA format and includes various groups including hofx and obsValue.

Running HofX3D#

First, you need to load the modules and set up the environment needed to run JEDI. More details available on Spack-Stack docs and JEDI docs

After loading the modules, set the path to your JEDI_BUILD and MPIEXEC. Running which mpiexec will return the path to your mpiexec. JEDI_BUILD is the path to your JEDI build directory.

On Discover#

On Discover, you can use Spack Stack Intel 1.9.0 and a a pre-build version of the JEDI-bundle:

Load Spack Stack Intel 1.9 on Discover by executing:

source /discover/nobackup/projects/gmao/advda/swell/jedi_modules/spackstack_1.9_intel

Point to a pre-build version of the code here: /discover/nobackup/projects/jcsda/s2127/maryamao/geos-esm/jedi-work/build-intel-release/bin

Here is an example of setting these two variables:

export MPIEXEC=/usr/local/intel/oneapi/2021/mpi/2021.10.0/bin/mpiexec
export JEDI_BUILD=/discover/nobackup/projects/jcsda/s2127/maryamao/geos-esm/jedi-work/build-intel-release/bin

All the files that you need are under hofx directory. So cd hofx and then you can run the application with this command:

$MPIEXEC "-n" "6" $JEDI_BUILD/fv3jedi_hofx_nomodel.x hofx_fv3-geos_aero_nomodel.yaml 2>&1 | tee log_hofx3d.txt

examining the output of HofX3D run#

In the previous section, we saved the output from running fv3jedi_hofx_nomodel.x in log_hofx3d.txt. The output log includes helpful information, such as timing statistics for all the tasks in the application. Examine the output log and take note of the tasks that used the most computational resource.

The output file name is specified under the obsdataout -> obsfile section of the YAML file. You can examine this file using nco command ncdump.

ncdump -h  output/fb.hofx3d.tempo_no2_tropo.20230805T150000Z.nc

The output file is in a nested NetCDF format (IODA format) and includes several groups such as MetaData, ObsValue, hofx, etc.

import xarray as xr
fb_file = xr.open_datatree('hofx/output/fb.hofx3d.tempo_no2_tropo.20230805T150000Z.nc')
fb_file

<xarray.DatasetView> Size: 72kB
Dimensions:   (Layer: 72, Location: 8895, Vertice: 73)
Coordinates:
  * Layer     (Layer) float64 576B 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
  * Location  (Location) float64 71kB 0.0 6.0 12.0 ... 8.989e+03 8.995e+03
  * Vertice   (Vertice) float64 584B 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
Data variables:
    *empty*
Attributes:
    Vertice:               73
    Location:              8998548
    converter:             tempo_nc2ioda.py
    _ioda_layout_version:  0
    nvars:                 1
    Layer:                 72
    sensor:                TEMPO
    platform:              Intelsat 40e
    _ioda_layout:          ObsGroup
    history:               Wed Oct 22 16:09:14 2025: ncks -d Location,1,89985...
    date_time_string:      1980-01-06T00:00:00Z
    NCO:                   netCDF Operators version 5.2.4 (Homepage = http://...

Ctest example#

In the fv3-jedi repository, in test/CMakeLists.txt look for various H(x) test examples. ctest fv3jedi_test_tier1_hofx_nomodel uses testinput/hofx_nomodel.yaml as the YAML input and fv3jedi_hofx_nomodel.x as the executable.

ecbuild_add_test( TARGET   fv3jedi_test_tier1_hofx_nomodel
                  MPI      6
                  ARGS     testinput/hofx_nomodel.yaml
                  COMMAND  fv3jedi_hofx_nomodel.x )

Compare testinput/hofx_nomodel.yaml with the YAML file described in the example above and note the differences.

Contents