Applying Filters in JEDI

Prerequisite¶

Please review here to learn about setting up your work environment and configuring the jedi-bundle.

Before starting, verify:

• $JEDI_BUILD environment variable is set

• fv3jedi_hofx_nomodel.x exists at $JEDI_BUILD/fv3jedi_hofx_nomodel.x

• Input files exist in the jedi_applications/run_hofx/hofx/inputs directory (background, observation, and auxiliary files such as geometry configurations)

• You have access to compute resources (interactive node or batch system)

• You have an understanding of how to run an HofX application in JEDI and have reviewed the HofX practical examples in this tutorial)

The input files for this practical example are available under jedi_applications/run_hofx/hofx/inputs

Introduction¶

In JEDI, the Quality Control (QC) filters are customizable and generic and are part of the UFO code repository. Filters are essential components in the data assimilation workflow that can modify the QC flags or observation error variances among other things. More about the filters in JEDI is available in the documentation.

This tutorial provides examples of how to use the JEDI filters in an HofX application. Note that filters are part of observation class and can be used for different applications such as Hofx, variational, etc.

In this example, for the TEMPO observation, we want to filter/flag pixels with Solar Zenith Angle (SZA) greater than 70 degrees and Cloud Fraction (CF) greater than 0.15.

First, examine the observation file to see how SZA and CF information are saved. You can find these variables under the dropdown menu below under Groups. Which group do they belong to?

YAML Structure¶

To filter out observations that meet the criteria we use RejectList filter. The YAML file below is similar to the HofX3D YAML except for the filter section under observations section.

...
observations:
  observers:
  - obs space:
      name: tempo_no2_tropo
      obsdatain:
        engine:
          obsfile: inputs/obs/tempo_no2_tropo_20230805T150000Z.nc
          type: H5File
      obsdataout:
        engine:
          allow overwrite: true
          obsfile: output/fb.hofx3d.filter.tempo_no2_tropo.20230805T150000Z.nc
          type: H5File
      observed variables:
      - nitrogendioxideColumn
      simulated variables:
      - nitrogendioxideColumn

    # filters are specified here
    obs filters:
    - filter: RejectList
      where:
      - variable:
          name: MetaData/cloud_fraction
        minvalue: 0.15
    - filter: RejectList
      where:
      - variable:
          name: MetaData/solar_zenith_angle
        minvalue: 70

    obs operator:
      name: ColumnRetrieval
      isApriori: false
      isAveragingKernel: true
      nlayers_retrieval: 72
      stretchVertices: topbottom
      tracer variables:
      - volume_mixing_ratio_of_no2

The complete input YAML is available under: hofx/yamls/hofx_fv3-geos_cf_nomodel_filters.yaml

Running hofx3d with filters¶

Now you can run the fv3jedi_hofx_nomodel.x application again to create a new feedback file fb.hofx3d.filter.tempo_no2_tropo.20230805T150000Z.nc that includes the QC flags. All the input files for this exercise are under jedi_applications/run_hofx/hofx/ directory. So cd this directory and then you can run the application with this command:

$MPIEXEC "-n" "6" $JEDI_BUILD/fv3jedi_hofx_nomodel.x yamls/hofx_fv3-geos_cf_nomodel_filters.yaml 2>&1 | tee log_hofx3d_filter.txt

On Discover, you will need to run this command on an interactive node or submit it as a Slurm job. Please see here for more details.

Review the output log file (log_hofx3d_filter.txt) to see how many observation points were rejected.

Examining the output¶

Compare the feedback file (output/fb.hofx3d.filter.tempo_no2_tropo.20230805T150000Z.nc) with the input observation file (inputs/obs/tempo_no2_tropo_20230805T150000Z.nc) and note the number of observations in each file. As you can see HofX values were computed for all the observation points regardless of the filter results. Filter sets the QC flags under QCEffective group in the feedback file with 0 for observation points that passed the filter and 14 for points that were filtered out. Based on the value of the QCEffective, JEDI will include/exclude the points in assimilation algorithm.