Additional errors from the post-processing scripts:

1. gfsfbwind
   gfsfbwind.log: ERROR config.resources must be sourced before sourcing WCOSS2.env

This error is triggered by the following logic statement within WCOSS.env:

if [[ -n "${ntasks:-}" && -n "${max_tasks_per_node:-}" && -n "${tasks_per_node:-}" ]]; then
    max_threads_per_task=$((max_tasks_per_node / tasks_per_node))
    NTHREADSmax=${threads_per_task:-${max_threads_per_task}}
    NTHREADS1=${threads_per_task:-1}
    [[ ${NTHREADSmax} -gt ${max_threads_per_task} ]] && NTHREADSmax=${max_threads_per_task}
    [[ ${NTHREADS1} -gt ${max_threads_per_task} ]] && NTHREADS1=${max_threads_per_task}
    APRUN_default="${launcher} -n ${ntasks}"
else
    echo "ERROR config.resources must be sourced before sourcing WCOSS2.env"
    exit 2
fi

The variable $tasks_per_node is currently absent from the 'fbwind' case module within config.resources, and will be added.

2. gfsgempakncdcupapgif
   gfsgempakncdcupapgif.log: End exgfs_atmos_gempak_gif_ncdc_skew_t.sh ... with error code 1

Preceding this error:

• the ImageMagick command that is invoked during the execution of make_tif.sh (convert) is not recognized
• the location of file make_NTC_file.pl is not recognized during the execution of make_tif.sh
• the subdirectory defined by $COM_OBS was found to be absent from the $ROTDIRS filespace

The file make_tif.sh is found to rely on the static reference to an outdated version of ImageMagick, and will need to be modified to reference a currently available system module. Further, make_tif.sh relies on the $UTILgfs variable to construct the path to make_NTC_file.pl. $UTILgfs is defined only within the job file $HOMEgfs/sorc/upp.fd/jobs/J_NCEPPOST, which references an antiquated 'util' directory that is no longer present within the UPP directory tree. Either or each of the files J_NCEPPOST or make_tif.sh will be modified to reflect the current path to make_NTC_file.pl.

Specific to the HR4 test of the global workflow, there may be a need to ensure proper staging of the observational data directory, defined by $COM_OBS. The workflow code will be reviewed for any need of revision.

These run to completion in CI tests (C96), albeit we are not validating the results.

@DavidHuber-NOAA Can you look at the env issues here?

@WalterKolczynski-NOAA The variable that @ChristopherHill-NOAA mentions not being in config.resources for the fbwind jobs is correct. That job is not run by us in the C96_atm3DVar_extended CI test. We should probably add it so it is tested.

For the gfsgempakncdcupapgif job, I see the same make_tif.sh and make_NTC_file.pl errors. However, the COM_OBS directory was found for our C96 test case. I suspect that the COM_OBS directory is not generated for a forecast-only experiment (assuming that is how this is run). In that case, the gfsgempakncdcupapgif should probably not be part of the mesh in forecast-only cases.

Going back to the original post, the C96_atm3DVar_extended test does not attempt to create the gfs_downstream.tar tarball, in which the WMO collective files would be stored. It's not immediately clear to me why not, as it should be triggered when DO_BUFRSND == YES, which it is. I will look into this.

We do not run the gfs_awips* or gfs_waveawips* jobs, so I don't have anything to compare against there. I wonder if the gfs_awips jobs should be run in the C96_atm3DVar_extended test. I know it adds a significant number of jobs.

AWIPS jobs should be running in the extended test. That's the main point of it. I believe fbwind is gated behind the AWIPS switch as well.

Traced back why it is was off. When I added the test in #2567, there was an issue with tocgrib2 and the convective precip fields, so AWIPS was turned off. That has since been fixed (#2652), so the test should be turned on now (it should've been turned on then).

Looking into the gfs_downstream.tar failure, I see that WCOSS2 does not enable HPSS archiving by default. Turning this feature on and attempting to run gfs_arch in the C96_atm3DVar_extended test causes an identical failure that Chris reported. HPSS archiving should probably be enabled during CI testing on WCOSS2.

I will address the gfs_downstream.tar issue in an upcoming PR.

The source of the missing postsnd files comes from #1929. This PR reworked the way data was sent to COM_ATMOS_WMO. Now, data is only sent to that directory if SENDDBN == "YES". However, the postsnd collective files are also (always) sent to COM_ATMOS_BUFR, though they are named differently. I will update the archive yaml to point to this location instead.

Pertaining to errors found within the gfsawips_20km_1p0deg_f###-f###.log files:

According to a relevant SCN from 2021, the 500-hPa 5-wave height (5WAVH) atmospheric field was to have been removed from all GFS GRIB2 products. As the script exgfs_atmos_awips_20km_1p0deg.sh invokes TOCGRIB2 to reference parameters from the parm/wmo/grib2_awpgfs* files, the absence of the expected field 5WAVH from the GRIB files results in "error code 30".

Given the above information, the 5WAVH entries will need to be removed from grib2_awpgfs_20km_[ak,conus,pac,prico]f000 and grib2_awpgfs[000-240].003 parameter files -- which should provide remedy to the error.

Additionally, 5WAVH is erroneously associated with the operation of a water wave model through a block of code (featured twice) within exgfs_atmos_awips_20km_1p0deg.sh:

      if [[ ${DO_WAVE} != "YES" ]]; then
         # Remove wave field it not running wave model
         grep -vw "5WAVH" "parm_list" > "parm_list_temp"
         mv "parm_list_temp" "parm_list"
      fi

The test run of GFS HR4 featured DO_WAVE=YES, and did not remove 5WAVH from parm_list. Nonetheless, the above code block was developed in error, and will need to be removed from exgfs_atmos_awips_20km_1p0deg.sh at lines 177-181 and 211-215.

The source of the missing postsnd files comes from #1929. This PR reworked the way data was sent to COM_ATMOS_WMO. Now, data is only sent to that directory if SENDDBN == "YES". However, the postsnd collective files are also (always) sent to COM_ATMOS_BUFR, though they are named differently. I will update the archive yaml to point to this location instead.

These issues were fixed in #3053.

@BoCui-NOAA @WalterKolczynski-NOAA @aerorahul
Should the fbwind executable be able to run on Hera? It compiled, but I am getting errors about being unable to read the index file. I will switch to WCOSS2 if not.

@BoCui-NOAA @WalterKolczynski-NOAA @aerorahul Should the fbwind executable be able to run on Hera? It compiled, but I am getting errors about being unable to read the index file. I will switch to WCOSS2 if not.

@DavidHuber-NOAA Are you referring to the executable fbwind from code sorc/gfs_utils.fd/src/fbwndgfs.fd?

@BoCui-NOAA Yes, that's correct.

If you would like to see a run log, it can be found here: /scratch1/NCEPDEV/stmp2/David.Huber/RUNDIRS/C96_atm3DVar_extended_extended/gfs.2021122100/fbwind.684074/OUTPUT.684369.

@DavidHuber-NOAA Unfortunately, I don't have experience with this code and compiling it on Hera.

Ah, OK, thank you.

@aerorahul Do you know who I should ping on this?

I think these are best debugged on wcoss2. These programs run into issues on RDHPCS platforms.

The buoy issue seems to be related to missing gfswave.46001.cbull file that should have been existed or created in upstream.

The buoy issue seems to be related to missing gfswave.46001.cbull file that should have been existed or created in upstream.

This is a known issue that we're working on a fix for from the model side: ufs-community/ufs-weather-model#2546 - I'll run a test w/this fix in the model so downstream can be re-tested.