r2d2_ingest suite

In the previous lecture, the hofx_cf workflow used GetObservations and GetBackground to stage its inputs. Those tasks assume that the requested files have already been saved and made available through R2D2. In this part of the training we provide a high level introduction to R2D2 and then introduce the two ingest configurations that populate it:

• Running the ingest_obs_cf configuration — downloading, converting, and ingesting observations.

• Running the ingest_background_cf configuration — registering existing GEOS-CF background files.

About R2D2¶

R2D2 (Research Repository for Data and Diagnostics) is a data catalogue. SWELL can ask R2D2 “give me the TEMPO NO₂ file for this window,” and R2D2 answers with the appropriate file.

R2D2 combines a metadata database with a storage location. The server records what data exists and where it is stored. When a workflow requests an item, the client asks the server where it is and then transfers the file from storage. Users and workflows do not need to know the filename or directory. They describe the file using R2D2 metadata keys.

The guiding idea is ingest once, use many times. SWELL’s R2D2 introduction has the full architecture diagram and the store/fetch reference. What follows is a high-level summary tailored for the CF application.

Items¶

Everything in R2D2 is an item, and the item type determines which keys identify it. If the keys are incorrect, the API may return a 400 error. Here is a list of R2D2 items:

These keys are used in commands such as r2d2.fetch, r2d2.store, and r2d2.search to fetch, save, and search for a specific file.

A few differences between items woth noting:

Observations have no experiment. They are shared input data and independent of a particular experiment. Once tempo_no2_tropo for a given provider and window is in R2D2, multiple experiments can fetch the same file.

Forecasts and feedback do have an experiment. These products are results from specific exeperiments with specific configurations, so the experiment key distinguishes among their files.

Example for item='observation':¶

import r2d2

# r2d2.fetch to get the data from R2D2 or r2d2.store to save the data on R2D2
r2d2.fetch(
  item='observation',
  observation_type='tempo_no2_tropo',
  provider='nasa_v4', # use this key to distinguish different versions of a retrieval, such as `nasa_v3` or `nasa_v4` for TEMPO
  window_start='20251015T210000Z',
  window_length='PT6H',
  target_file='obs_tempo_no2_tropo.nc4', # save the file under target_file name
  file_extension='nc4'
  )

Note, that typically observtions are grouped based on 6h assimilation windows with starting point at 21Z, 03Z, 09Z, and 15Z. window_start indicates the start of the assimilation window and window_length indicates the length of the assimilation window. In theory, nothing prevents users to change the window_length to a shorter or longer assimilation window or to select a different window_start. For GEOS-CF case, we typically use 6h windows with conventional starting point of 21Z, 03Z, 09Z, and 15Z.

Example for item='forecast':¶

import r2d2

# r2d2.fetch to get the data from R2D2 or r2d2.store to save the data on R2D2
r2d2.fetch(
  item='forecast',
  model='geos_cf', # other options include cice6 and mom6
  experiment='swell_test', #r2d2_experiment_id
  resolution='c90',
  step='PT3H', # forecast lead time or the elapsed duration from the forecast initialization time (`date`). Forecast valid_time = date + step 
  date='2023-08-05T09:00:00Z',
  file_extension='nc', # this is not a very consistent key and may be depricated at some point
  target_file = 'bkg.20230805T180000Z.nc4'

  )

Note: For window_start or date, use a UTC ISO 8601 string (YYYY-MM-DDTHH:MM:SSZ). R2D2 also accepts other separator styles by stripping non-digit characters, but using this format consistently is recommended. For step make sure you use ISO 8601 such as PT0H, PT6H, or P1D.

Where the files actually live: databases available to SWELL¶

So far we have described how to name a file. The other half of the picture is where the file is kept. Recall that R2D2 splits metadata from storage: the API server holds the catalogue, and the files themselves sit somewhere else. R2D2 organizes those storage locations in three levels:

When you run on Discover, your compute host is registered against a data hub, and that hub contains one or more data stores. You normally do not set any of this. If r2d2_datastore is left empty in experiment.yaml, R2D2 picks the highest-priority writable store for your compute host, and that is the right answer almost all of the time. Set it only when you deliberately need to write to or read from a particular store:

r2d2_datastore: r2d2-experiments-nccs-gmao # optional; leave empty to let R2D2 choose

To see what is available from your machine, SWELL ships a discovery script that prints your client context, the registered compute hosts, all data stores, and which of them you can write to:

python src/swell/utilities/scripts/discover_r2d2_datastores.py --platform nccs_discover_sles15

Separately from the data stores, there is more than one R2D2 server (catalogue): GMAO (still in testing phase) and JCSDA each run their own, and they hold different data. Which one you talk to is set by your credentials, see Getting Started. SWELL also accepts a ~/.swell/r2d2_credentials.yaml file with named profiles instead of environment variables, in which case r2d2_server: gmao_server in experiment.yaml selects the profile.

A few other things worth knowing¶

• You need the R2D2 client module. R2D2 is not part of the SWELL package. On Discover it comes from module load r2d2-client/112025, which SWELL loads for you when a task needs it. See here.

• experiment must be registered before you can store to it. Observations do not use the experiment key, but forecasts, analyses, and feedback do, and R2D2 will reject a store to an unknown experiment. This is why SWELL keeps a separate r2d2_experiment_id, distinct from your local experiment_id, and appends a random hex suffix if the name is already taken.

• r2d2.search before you ingest. Searching with a partial set of keys tells you what is already in the catalogue, which is the quickest way to check whether someone has already ingested the window you are about to work on. This matters because the whole point is ingest once, use many times.

• Storing does not always copy. store_as_symlink: true registers a symlink to a file that is already on a shared filesystem instead of duplicating it, which is what Running the ingest_background_cf configuration does for GEOS-CF backgrounds.

• dry_run: true is your friend. Every SWELL ingest task honors it and will log the R2D2 operations it would perform without writing anything. Use it first.

R2D2 ingest suites in SWELL¶

The names presented to users by swell create are suite configurations. Internally, multiple configurations can be listed in one suite_config.py. For example, the r2d2_ingest suite has this structure:

src/swell/suites/r2d2_ingest/           ← the suite (one flow.cylc)
├── flow.cylc
└── suite_config.py
    ├── r2d2_ingest                     ← base configuration
    ├── ingest_obs_marine               ← marine observation configuration
    ├── ingest_obs_cf                   ← CF observation configuration
    └── ingest_background_cf            ← CF background configuration

Therefore, ingest_obs_cf and ingest_background_cf are not separate suite directories. They are two configurations of r2d2_ingest, and they select different branches of the same flow.cylc using the download_convert_pipeline and ingest_background_pipeline flags. Two switches determine which branch of that graph runs:

• download_convert_pipeline — when True, run download + convert + ingest (this is what ingest_obs_cf sets). This branch is covered in Running the ingest_obs_cf configuration.

• ingest_background_pipeline — when True, run the background-saving branch instead (used by ingest_background_cf). This branch is covered in Running the ingest_background_cf configuration.

Next steps¶

The next two lectures walk through two CF ingest workflows:

• Running the ingest_obs_cf configuration — the ingest_obs_cf configuration downloads raw observation files from a remote server, converts them to IODA format with the ioda-converters, and stores the IODA files in R2D2 as observation items.

• Running the ingest_background_cf configuration — the ingest_background_cf configuration registers GEOS-CF background files that already exist on the CSS shared filesystem in R2D2 as forecast items, normally as symlinks rather than copies.