JEDI-based ensemble recentering and analysis calculation

[DavidNew-NOAA]

Description

COORDINATED MERGE

This PR implements ensemble recentering and analysis calculation in the Global Workflow, using JEDI-based applications to replace certain GSI utilities when JEDI is turned on in the workflow. If using GSI, then the workflow will remain unchanged. This PR also (finally) implements native-grid DA increments into the worflow.

The gdas_analcalc and enkfgdas_ecen jobs will be replaced by gdas_analcalc_fv3jedi and enkfgdas_ecen_fv3jedi jobs respectively. The enkfgdas_echgres job is eliminated, since changing of resolution of the deterministic backgrounds is done internally in the JEDI-based recentering application.

The design for this PR is based on discussions between the DA team and GW team a few months ago. Explanation of the flow of data through the workflow:

The gdas_analcalc_fv3jedi job dependencies do not change. The native-grid backgrounds andincrements are staged, and then the GDASApp JEDI fv3jedi_add_increments application is run to add them and interpolate to the Gaussian grid. The Gaussian-grid backgrounds are also staged, and then a simple Python function inserts to these analysis variables into the histories, which become the Gaussian analyses. This is done this way to guarantee that the resulting Gaussian analyses are in the exact format required by UPP.

The enkfgdas_ecen_fv3jedi no longer depends on the analysis calc job, since the ensemble-resolution variational analysis is computed/interpolated internally in the JEDI-based recentering application. All other job dependencies remain the same. We no longer need to compute the ensemble mean analysis in this job, since it can be outputted the the JEDI local ensemble DA application in the enkfgdas_atmensanlsol job and just staged for recentering. The variational increment and deterministic backgrounds are also staged to compute the ensemble-resolution variational analysis. The output of this job is no longer the recentered ensemble increments, but rather the "correction increment", which when added to ensemble increments becomes the recentered increments. The prefix for the "correction increment" is catminc.

The enkfgdas_fcst job now stages both the ensemble increments and the correction increment. They are added together with ncbo in forecast_postdet.sh to generate the recentered increment.

All forecast increments, both deterministic and ensemble, are now on the native cubed-sphere grid

Resolves #3248

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES
• Does this change require a documentation update? IDK
• Does this change require an update to any of the following submodules? YES
  • EMC verif-global
  • GDAS #1488
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

• Clone and build on Hera
• Run C96C48_ufs_hybatmDA

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidNew-NOAA]

The remaining shell norm warnings appear to be false positives

[RussTreadon-NOAA]

@DavidNew-NOAA , I cloned DavidNew-NOAA:feature/calcanl on Hera. I added the following to JGDAS_ENKF_ECEN_FV3JEDI

@@ -6,7 +6,8 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "ecen_fv3jedi" -c "base ecen_fv3jedi"
 ##############################################
 # Set variables used in the script
 ##############################################
-
+# Ignore possible spelling error (nothing is misspelled)
+# shellcheck disable=SC2153
 GDATE=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} - ${assim_freq} hours")
 gPDY=${GDATE:0:8}
 gcyc=${GDATE:8:2}

There were no shellcheck warnings for JGDAS_ENKF_ECEN_FV3JEDI after adding these lines.

[DavidNew-NOAA]

@RussTreadon-NOAA Thank you for the tutorial on fixing erroneous shell norms! It took a few iterations, but I've figured it out

[RussTreadon-NOAA]

Interesting. I updated to d9e0275.

Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$ git log --oneline | head -1
d9e0275a Shell norm fix
Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$ shellcheck jobs/JGDAS_ENKF_ECEN_FV3JEDI
Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$

No shellcheck warnings.

I am using a copy of shellcheck which originates from @WalterKolczynski-NOAA

Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$ which shellcheck
/home/Rahul.Mahajan/bin/shellcheck
Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$
Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$ ls -l /home/Rahul.Mahajan/bin/shellcheck
lrwxrwxrwx 1 Rahul.Mahajan da 64 Feb 26  2023 /home/Rahul.Mahajan/bin/shellcheck -> /scratch2/NCEPDEV/ensemble/save/Walter.Kolczynski/bin/shellcheck

Is this version of shellcheck too old?

I see that g-w has a .shellcheckrc. I reran shellcheck ingoring the .shellcheckrc file

Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/calcanl$ shellcheck --norc jobs/JGDAS_ENKF_ECEN_FV3JEDI

In jobs/JGDAS_ENKF_ECEN_FV3JEDI line 3:
source "${HOMEgfs}/ush/preamble.sh"
       ^--------------------------^ SC1091 (info): Not following: ./ush/preamble.sh was not specified as input (see shellcheck -x).
        ^--------^ SC2154 (warning): HOMEgfs is referenced but not assigned.


In jobs/JGDAS_ENKF_ECEN_FV3JEDI line 4:
source "${HOMEgfs}/ush/jjob_header.sh" -e "ecen_fv3jedi" -c "base ecen_fv3jedi"
       ^-----------------------------^ SC1091 (info): Not following: ./ush/jjob_header.sh was not specified as input (see shellcheck -x).


In jobs/JGDAS_ENKF_ECEN_FV3JEDI line 12:
GDATE=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} - ${assim_freq} hours")
                                        ^----^ SC2154 (warning): cyc is referenced but not assigned (did you mean 'gcyc'?).
                                                 ^-----------^ SC2154 (warning): assim_freq is referenced but not assigned.


In jobs/JGDAS_ENKF_ECEN_FV3JEDI line 47:
if [[ -e "${pgmout}" ]] ; then
          ^-------^ SC2154 (warning): pgmout is referenced but not assigned.

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- HOMEgfs is referenced but not ass...
  https://www.shellcheck.net/wiki/SC1091 -- Not following: ./ush/jjob_header....

Now I see shellcheck warnings ... but these warnings are not the ones we see from github actions.

Any ideas @WalterKolczynski-NOAA, @aerorahul , @KateFriedman-NOAA as to (a) what's going on and (b) how to get the github shellcheck action to pass?

[DavidNew-NOAA]

@aerorahul @WalterKolczynski-NOAA This PR is ready for review. It has a pending companion GDAS PR, but I'd like some feedback on this PR before I merge the GDAS PR and break things.