BUFR2IODA Conversion

[Dooruk]

The following lays out steps required for us to incorporate the capabilities of having swell convert bufr files to IODA on the fly. This is an edited version of the original written by @Dooruk. I will try to put some names around the items below to sort of give an idea of who might be best to work on a given item, but please take those as suggested names. If anyone feels that's not quite something within their realm, no problem, just let us know.

• GEOSadas: move obsys.rc to sidb in mksi (Ricardo) - GEOS_mksi tag: g5.41.2
• GEOSadas: pull acquire applications into a repo of their own - see below (Ricardo)
• SWELL needs to learn how to extract the obs classes available and applicable to a given cycle time; that is, given a set of obs-classes, SWELL should read from the present databased (obsys.rc), extract the class of interest, see if data is available for the date/time of request, and create a list of fully resolve file names from the database that can then be passed on the either (or both) of the step below. (Ricardo) - Akira and R had a conversation about this. R will bring the GEOSDAS machinery to bring observations into the run experiment given a series of obsclass, a date and a couple of other parameters. This capability is based on a series of Perl scripts; they will require the installation in SWELL to know than simply handling Python scripts. I will need to work with Michael and Alexey on this. The command line to bring in data into the run env of converter suite will be as simple as the following:

acquire_obsys -v -d $FVWORK -s $SPOOL -la $nlook -ssh \ -strict $nymd $nhms $nfreq $nstep $OBSCLASS

where OBSCLASS defines the classes, as for example: disc_airs_bufr,disc_amsua_bufr,gmao_amsr2_bufr
nymd and nhms are the GEOS convention for 8-digit date and 6-digit time, and I will explain the other variables when I add the PR to that will allow swell to pull the required codes the an adequate ESM-repo.

The software needed to allow swell to invoke a command line similar to what's written about has now been extract from GEOS and placed in a repo of its own: https://github.com/GEOS-ESM/GMAO_perllib ; SWELL needs to be modified to:

• when cloned, also clone GMAO_perllib - please do not pull develop/main, the following tag should be used: g1.0.1
• when installed, also install GMAO_perilib (see CMakelist in GMAO_perllib to see how files and programs need to fall into the bin directory; notice that with this, SWELL will need to be extended to learn to install things that are not python. I don't know pip very well, but I believe that is a python-centric and therefore, I believe the install procedures of SWELL will need to be generalized.

There are two options to implement the converters:

1. Offline converter: once SWELL the steps above are complete:

• BUFR Offline conversion/storage #454

• build infrastructure in SWELL to process (go over) each of the files in the list created above, and apply the corresponding converter. In some case, as I understand, the convert program will be different (GPSRO seems to be a case in point), so SWELL will need to know which application to call for a given class. (Seth, Akira, Doruk, Others could contribute to this, esp in the arrangement of yamls.)

• the conversion should try to create IODA files in the form presently created by ncdiag2ioda; this might be a little trick for conventional (prepbfr) data, but it seems doable. (we can ALL think together about this)

• There could be a suite in SWELL associated with the converter with the function of populating R2D2 once the converted files are available; this could be a suite ran only by a few people (those w/ the ability to write in R2D2) - or, for testing purposes, a user should be able to create its own temperature R2D2 to live somewhere under its own file systems). (Alexey, Doruk, Seth, ?)

• the converter step should also be an option implemented within the Var suites so that these could, in principle, process data that might not have been process and put in R2D2. (Ricardo, Doruk, Alexey)

• The part that puts together the observation yamls, to make up the final (Var) yaml, will need to be adapted to include data thinning, super-obbing, additional QC; whatever needs to be done w/ the data before the HOFX sees the data. Ideally these will be a separate set of yaml files associated w/ each ob-space that is handled by SWELL as it comes up w/ the final yaml.

2. Online converter:

This is a set that would bypass the creation of the IODA files (converted from bufr), allowing the Var to run directly from the original bufr files. This might be a good option for operations, to avoid the cost of reading the data twice. This should be relatively simple to put together once the steps above for the offline processing have been completed. I will suggest postponing this one until we gather some experience and get meaningful results running w/ what the offline procedures will generate.

I think some of the stuff below this line is outdated ... @cohen-seth has made considerable progress processing various data classes. I'll leave it up to others to edit/update/remove.

========================================================================================

BUFR file locations:

/discover/nobackup/projects/gmao/input/dao_ops/ops/flk/ncep_g5obs/bufr

⚠️ Before starting

Make sure bufr2ioda.x is in the $PATH, the proper modules are loaded, and dependencies (for instance, $PYTHONPATH and pyiodaconv) are created.

Observers:

• amsua_aqua (ioda-converters/test/testinput/bufr_amsua.yaml)

Example file format: gdas1.211210.t00z.1bamua.tm00.bufr_d

bufr2ioda.x testinput/bufr_ncep_adpsfc.yaml" gdas.t12z.adpsfc.tm00.nc

• amsua_metop-b
• amsua_metop-c
• amsua_n15
• amsua_n18
• amsua_n19
• atms_n20
• atms_npp
• avhrr3_metop-b
• avhrr3_n18
• avhrr3_n19
• cris-fsr_n20
• cris-fsr_npp
• gmi_gpm

FYI @elakkraoui

[cohen-seth]

Bufr to Ioda conversion tests results (offline):

The only changes needed were to change the paths in the input and output lines in the existing yaml files (test folder).

classes	converter	yaml	result
1bmhs	bufr2ioda.x	bufr_ncep_1bmhs.yaml	successful	✅
amsua	bufr2ioda.x	bufr_ncep_1bamua_n15.yaml	successful	✅
amsua	bufr2ioda.x	bufr_ncep_1bamua_ta.yaml	successful	✅
atms	bufr2ioda.x	bufr_ncep_atms.yaml	successful	✅
mtiasi	bufr2ioda.x	bufr_ncep_mtiasi.yaml	successful	✅
satwind (am & pm)	bufr2ioda.x	bufr_ncep_satwind_avhrr.yaml	successful	✅
aircft	bufr2ioda.x	bufr_ncep_prepbufr_aircft.yaml	successful	✅
sevcsr	bufr2ioda.x	bufr_ncep_sevcsr.yaml	successful	✅