Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUFR Offline conversion/storage #454

Open
Tracked by #378
Dooruk opened this issue Nov 4, 2024 · 0 comments
Open
Tracked by #378

BUFR Offline conversion/storage #454

Dooruk opened this issue Nov 4, 2024 · 0 comments
Assignees
Labels
BUFR BUFR related work (convert, store, enjoy) enhancement New feature or request

Comments

@Dooruk
Copy link
Collaborator

Dooruk commented Nov 4, 2024

The detailed steps required for this are outlined in #378. I break it down here as bite size tasks and provide a little more context for R2D2 and adding a suite. This is not an easy task so @cohen-seth, please ask our help if any of this is not clear or if you are stuck.

  1. Create a tarball with BUFR files that need to be converted located in /discover/nobackup/projects/gmao/input/dao_ops/ops/flk/ncep_g5obs/bufr for testing. We need two sets of files for 2021-12-12T00:00:00Z and 2021-12-12T06:00:00Z for x0048 for now. We will then need a new task called get_bufr.py that copies these files to the SWELL cycle directory and extracts them.

  2. Create BUFR yaml files defined in here using GMAO settings in src/swell/configuration/jedi/interfaces/geos_atmosphere/observations. They could reside in a folder under observations called bufr for now.

  3. For the offline converter, we need a new suite. convert_ncdiags is the exact template we should replicate; for our case, at minimum, we need get_bufr.py and bufr_to_ioda.py. bufr_to_ioda.py converts and stores to R2D2 (see below) and will execute bufr2ioda.x.

If bufr_to_ioda.py requires compute nodes it could be separated into separate convert and store tasks. save_obs_diags.py uses store API, @asewnath modified that one, she could answer if there are any questions.

Details:

  • Using R2D2 for input is not necessary as we will be using obsys*.rc files located in here.

  • R2D2 utilizes two locations while using fetch (obtaining files from the database) and store (save files from cycle directory to database) methods:
    Shared: /discover/nobackup/projects/gmao/advda/R2D2DataStore/Shared
    Local: Defined in experiment.yaml by the r2d2_local_path key. For simplicity a similar structure should be followed, for instance I use: /discover/nobackup/dardag/R2D2DataStore/Local.
    Shared locations for platforms are defined here.

  • Adding a new suite is as easy as copying convert_ncdiags to a new folder and manipulating the flow.cylc file. The tricky part is manipulating task_questions.yaml according to required keys that could be utilized for a certain task. For instance get_bufr task will require a key in task_questions.yaml pointing to the BUFR tarball described in step 1.
    A similar structure is used for suite_questions.yaml which controls which suites can access which suite configuration inputs.

  • It is possible to visualize Cylc flow.cylc files using the following command (for 3dvar suite) AFTER succesfully running the swell create command and creating an experiment folder (e.g., 3dvar-suite for the below example):
    cylc graph -c -o 3dvar_graph.png ./swell-3dvar/3dvar-suite/

FYI @ashiklom @gmao-yzhu

@Dooruk Dooruk mentioned this issue Nov 4, 2024
25 tasks
@Dooruk Dooruk added enhancement New feature or request BUFR BUFR related work (convert, store, enjoy) labels Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BUFR BUFR related work (convert, store, enjoy) enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants