Add GEFS regression test suite from EP5r2 configuration/case

[NickSzapiro-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR updates the cpld_bmark_p8 tests to a prototype GEFS test case of fully coupled s2swa+IAU+stochastics in atmosphere and ocean, with configuration and warm starts from restarts of EP5r2 ensemble member 1 for 2021-03-25 06Z. The EP5r2 test case was kindly provided by @bingfu-NOAA via @junwang-noaa with aerosol input data and configurations from @lipan-NOAA.

A separate INPUTDATA_ROOT_BMIC is no longer needed and is removed.

The regression test suite samples basic reproducibility/quality checks, particularly:

• control reproduces itself
• restart reproduces control
• changing number of tasks reproduces control
• changing number of threads reproduces control
• Intel debug version reproduces itself
• GNU debug version reproduces itself

All tests do not pass across all platforms, summarized in this regression test suite matrix:

Failures may require library/platform support. Hopefully committing this test suite as work in progress facilitates progress, and work will continue in follow up issues, including:

• Larger issue of GNU support
• Sensitivity to spack-stack updates
• Individual test failures

Note that there are two intentional differences from GEFS workflow configuration (please inform if you see other differences): 1) aerosols are 1-way coupled in diagnostic mode and 2) wave element mask has been modified as discussed in NOAA-EMC/WW3#1328.

In the future, depending on aerosol coupling, GOCART .rc files and ExtData directory structure may be revised for consistency with global-workflow. This benchmark configuration and case may be updated as well, particularly with GEFS reforecast or UFS case study.

TODO: Scripts need finalizing once filepaths are in shared space.
Input data is currently in user space on hera:
/scratch1/NCEPDEV/nems/Nick.Szapiro/tasks/input_data/gefs.v13/RT_GEFS/

Commit Message:

* UFSWM - Add GEFS regression test suite from EP5r2 configuration/case

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Update cpld_bmark_p8 with GEFSv13 EP5 configuration #1987
• Closes Add RT test for gocart_on, gccpp_on, nasa_on #1818

Sub component Pull Requests:

• None

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.

Input data Changes:

• New input data.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[NickSzapiro-NOAA]

Hi @jkbk2004 . If there are no input data changes coming up, can we stage input data for new test cases? Maybe @[INPUTDATA_ROOT]/GEFS/ is a good place.

It's on hera at /scratch1/NCEPDEV/nems/Nick.Szapiro/tasks/input_data/gefs.v13/RT_GEFS/ . The contents of WW3 subdirectory probably belong under INPUTDATA_ROOT_WW3 instead

[edwardhartnett]

Are the versions of PIO, netcdf-fortran, netcdf-c, and HDF5 all the same on hercules as on other platforms?

[NickSzapiro-NOAA]

Are the versions of PIO, netcdf-fortran, netcdf-c, and HDF5 all the same on hercules as on other platforms?

Hi @edwardhartnett. Hercules is maybe the only platform reliably running gnu tests right now. I see the same gnu and intel libraries via spack-stack-1.6.0 on hercules:
gcc/12.2.0: parallelio-2.5.10 parallel-netcdf-1.12.2 netcdf-fortran-4.6.1 netcdf-c-4.9.2 hdf5-1.14.0
intel/2021.9.0: parallelio-2.5.10 parallel-netcdf-1.12.2 netcdf-fortran-4.6.1 netcdf-c-4.9.2 hdf5-1.14.0

[edwardhartnett]

OK, I thought we had switched to hdf5-1.14.3?

Some parallel I/O bugs were fixed in HDF5-1.14.3.

[NickSzapiro-NOAA]

OK, I thought we had switched to hdf5-1.14.3?

Some parallel I/O bugs were fixed in HDF5-1.14.3.

@edwardhartnett We can test with updated packages. What versions would you suggest?

[edwardhartnett]

hdf5-1.14.3