Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove 5WAVH from AWIPS GRIB2 parm files #3146

Conversation

ChristopherHill-NOAA
Copy link
Contributor

@ChristopherHill-NOAA ChristopherHill-NOAA commented Dec 6, 2024

Description

As referred within #3019, the variable 5WAVH is being removed from each of the files parm/wmo/grib2_awpgfs[000-240].003 and parm/wmo/grib2_awpgfs_20km_[ak,conus,pac,prico]f000 for the purpose of remedying "error code 30" that was generated through the execution of exgfs_atmos_awips_20km_1p0deg.sh during the GFSv17 HR4 test run. Obsolete code is also being removed from the script exgfs_atmos_awips_20km_1p0deg.sh.

No other errors mentioned in #3019 are addressed in this PR.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO
    (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

Removal of variable 5WAVH from the GRIB2 files should allow completion of TOCGRIB2 processing (within exgfs_atmos_awips_20km_1p0deg.sh) of the GRIB2 files. @RuiyuSun, or the GW team, may wish to include the requested modifications for future GFSv17 tests that include post-processing jobs.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Issue #3019 - removing 5WAVH from AWIPS GRIB2 parm files _ Remove 5WAVH from AWIPS GRIB2 parm files Dec 9, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

WalterKolczynski-NOAA commented Dec 10, 2024

The code changes look fine, though I don't understand why they are needed. The code removed should have already stripped out 5WAV when running without waves. Is the 5WAV field no longer produced when waves are on? Need confirmation from VPPPG that this product is no longer needed.

Nevermind, just read more comments in the source issue.

@aerorahul
Copy link
Contributor

@GwenChen-NOAA Would you mind giving this PR a review? We would like to merge this soon.

@GwenChen-NOAA
Copy link
Contributor

GwenChen-NOAA commented Dec 11, 2024 via email

@aerorahul
Copy link
Contributor

@ChristopherHill-NOAA @GwenChen-NOAA would like to view the output of the changes to validate and review this PR

@ChristopherHill-NOAA
Copy link
Contributor Author

@aerorahul @GwenChen-NOAA I am staging the global workflow code in WCOSS2 with the HR4 model output (from 2020103000), and I will attempt to replicate the HR4 post-processing jobs with a test of the PR changes.

@RuiyuSun
Copy link

@GwenChen-NOAA would like to view the output of the changes to validate and review this PR

@GwenChen-NOAA
Copy link
Contributor

GwenChen-NOAA commented Dec 19, 2024 via email

@ChristopherHill-NOAA
Copy link
Contributor Author

@RuiyuSun and @GwenChen-NOAA : I was unable to complete a test of workflow post-processing jobs before WCOSS2 machine Dogwood was taken offline. I have been attempting to stage the global workflow code and input data on RDHPCS machine Hera to test PR changes there. This effort may be simpler to complete for someone who has recently tested global workflow code on either WCOSS2 or RDHPCS.

The code revisions included with this PR can be used to generate test data and to verify whether the "error code 30" from exgfs_atmos_awips_20km_1p0deg.sh is alleviated.

@RuiyuSun
Copy link

I won't be able to make a run until WCOSS2/dogwood comes back either.

@RuiyuSun
Copy link

I conducted a test on Dogwood using the latest workflow, develop branch, and a HRv4 IC(2020060100). The following jobs failed.
gfs_gempakmeta,
most of gfs_awips_20km_1p0deg_* (gfs_awips_20km_1p0deg_f237-f237 was successful),
gfs_fbwind

gfs_gempakgrb2spec* jobs didn't get started.

Please see below for the experiment dirs:
global workflow: /lfs/h2/emc/physics/noscrub/ruiyu.sun/git/global_workflow.git.dev4product
The experiment dir: /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/Devproducts
ROTDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/ROTDIRS/Devproducts
RUNDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/RUNDIRS/Devproducts

@RuiyuSun
Copy link

I am running a test based on Chris's branch and will post result here.

@DavidHuber-NOAA
Copy link
Contributor

@RuiyuSun @ChristopherHill-NOAA I have made a few fixes to the downstream jobs in #3187 that also address issues indicated in #3019.

@RuiyuSun
Copy link

@DavidHuber-NOAA Great! Thanks for the update.

@RuiyuSun @ChristopherHill-NOAA I have made a few fixes to the downstream jobs in #3187 that also address issues indicated in #3019.

@RuiyuSun
Copy link

I am running a test based on Chris's branch and will post result here.

The test finished with a few failures. Please see below for the failed tasks:

gfs_gempakncdcupapgif (failed)
gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs)
gfs_fbwind (failed)
gfs_waveawipsbulls (failed)
gfs_waveawipsgridded(failed)

Information about the test:
global workflow: /lfs/h2/emc/physics/noscrub/ruiyu.sun/git/global_workflow.chris ( features/iss3019_HR4_awips20 branch)
Expdir : /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD
ROTDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/ROTDIRS/HRv4PROD (log files)
RUNDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/RUNDIRS/HRv4PROD

@RuiyuSun
Copy link

RuiyuSun commented Jan 7, 2025

I am running a test based on Chris's branch and will post result here.

The test finished with a few failures. Please see below for the failed tasks:

gfs_gempakncdcupapgif (failed) gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs) gfs_fbwind (failed) gfs_waveawipsbulls (failed) gfs_waveawipsgridded(failed)

Information about the test: global workflow: /lfs/h2/emc/physics/noscrub/ruiyu.sun/git/global_workflow.chris ( features/iss3019_HR4_awips20 branch) Expdir : /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD ROTDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/ROTDIRS/HRv4PROD (log files) RUNDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/RUNDIRS/HRv4PROD

@ChristopherHill-NOAA Is this good enough to prove that your changes worked? Do we need anything else to proceed with this?

@ChristopherHill-NOAA
Copy link
Contributor Author

I am running a test based on Chris's branch and will post result here.

The test finished with a few failures. Please see below for the failed tasks:
gfs_gempakncdcupapgif (failed) gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs) gfs_fbwind (failed) gfs_waveawipsbulls (failed) gfs_waveawipsgridded(failed)
Information about the test: global workflow: /lfs/h2/emc/physics/noscrub/ruiyu.sun/git/global_workflow.chris ( features/iss3019_HR4_awips20 branch) Expdir : /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD ROTDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/ROTDIRS/HRv4PROD (log files) RUNDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/RUNDIRS/HRv4PROD

@ChristopherHill-NOAA Is this good enough to prove that your changes worked? Do we need anything else to proceed with this?

@RuiyuSun According to the log file /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD/logs/2020060100.log, each of the gfsawips_20km_1p0deg tasks ran to successful completion. This is indication that the code changes associated with this pull request generated the intended result.

I was not able to verify the content of the gfsawips_20km_1p0deg output files before they were scrubbed from ptmp space. Please let me know if these output files were saved in HPSS or elsewhere.

The failures associated with gfs_gempakncdcupapgif, gfs_gempakgrb2spec, gfs_fbwind, gfs_waveawipsbulls, and gfs_waveawipsgridded will be handled through separate pull requests.

@RuiyuSun
Copy link

RuiyuSun commented Jan 7, 2025

I am running a test based on Chris's branch and will post result here.

The test finished with a few failures. Please see below for the failed tasks:
gfs_gempakncdcupapgif (failed) gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs) gfs_fbwind (failed) gfs_waveawipsbulls (failed) gfs_waveawipsgridded(failed)
Information about the test: global workflow: /lfs/h2/emc/physics/noscrub/ruiyu.sun/git/global_workflow.chris ( features/iss3019_HR4_awips20 branch) Expdir : /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD ROTDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/ROTDIRS/HRv4PROD (log files) RUNDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/RUNDIRS/HRv4PROD

@ChristopherHill-NOAA Is this good enough to prove that your changes worked? Do we need anything else to proceed with this?

@RuiyuSun According to the log file /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD/logs/2020060100.log, each of the gfsawips_20km_1p0deg tasks ran to successful completion. This is indication that the code changes associated with this pull request generated the intended result.

I was not able to verify the content of the gfsawips_20km_1p0deg output files before they were scrubbed from ptmp space. Please let me know if these output files were saved in HPSS or elsewhere.

The failures associated with gfs_gempakncdcupapgif, gfs_gempakgrb2spec, gfs_fbwind, gfs_waveawipsbulls, and gfs_waveawipsgridded will be handled through separate pull requests.

@ChristopherHill-NOAA I didn't see any failure with awips jobs. All the failed jobs are listed below.

gfs_gempakncdcupapgif (failed)
gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs)
gfs_fbwind (failed)
gfs_waveawipsbulls (failed)
gfs_waveawipsgridded(failed)

@RuiyuSun
Copy link

RuiyuSun commented Jan 7, 2025

I am running a test based on Chris's branch and will post result here.

The test finished with a few failures. Please see below for the failed tasks:
gfs_gempakncdcupapgif (failed) gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs) gfs_fbwind (failed) gfs_waveawipsbulls (failed) gfs_waveawipsgridded(failed)
Information about the test: global workflow: /lfs/h2/emc/physics/noscrub/ruiyu.sun/git/global_workflow.chris ( features/iss3019_HR4_awips20 branch) Expdir : /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD ROTDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/ROTDIRS/HRv4PROD (log files) RUNDIRS: /lfs/h2/emc/ptmp/ruiyu.sun/RUNDIRS/HRv4PROD

@ChristopherHill-NOAA Is this good enough to prove that your changes worked? Do we need anything else to proceed with this?

@RuiyuSun According to the log file /lfs/h2/emc/physics/noscrub/ruiyu.sun/para_gfs/HRv4PROD/logs/2020060100.log, each of the gfsawips_20km_1p0deg tasks ran to successful completion. This is indication that the code changes associated with this pull request generated the intended result.
I was not able to verify the content of the gfsawips_20km_1p0deg output files before they were scrubbed from ptmp space. Please let me know if these output files were saved in HPSS or elsewhere.
The failures associated with gfs_gempakncdcupapgif, gfs_gempakgrb2spec, gfs_fbwind, gfs_waveawipsbulls, and gfs_waveawipsgridded will be handled through separate pull requests.

@ChristopherHill-NOAA I didn't see any failure with awips jobs. All the failed jobs are listed below.

gfs_gempakncdcupapgif (failed) gfs_gempakgrb2spec_f??? (don't started, perhaps due to dependence to other failed jobs) gfs_fbwind (failed) gfs_waveawipsbulls (failed) gfs_waveawipsgridded(failed)

Unfortunately, I didn't save the data.

@DavidHuber-NOAA
Copy link
Contributor

@ChristopherHill-NOAA @RuiyuSun I am OK with merging this when you deem it ready and I will continue the effort in #3187.

@ChristopherHill-NOAA
Copy link
Contributor Author

@ChristopherHill-NOAA @RuiyuSun I am OK with merging this when you deem it ready and I will continue the effort in #3187.

@DavidHuber-NOAA Since the targeted workflow tasks were observed to run to successful completion with the PR changes during the test run by @RuiyuSun, we agree with proceeding with the merge of this PR.

@DavidHuber-NOAA DavidHuber-NOAA merged commit f1afac9 into NOAA-EMC:develop Jan 7, 2025
5 checks passed
tsga added a commit to tsga/global-workflow that referenced this pull request Jan 22, 2025
* develop:
  Add echgres as a dependency only for RUN=enkfgdas, not enkfgfs (NOAA-EMC#3246)
  Add domain level to wave gridded COM path (NOAA-EMC#3137)
  CI JJOB Tests using CMake (NOAA-EMC#3214)
  Make assorted updates to waves (NOAA-EMC#3190)
  Move WCOSS2 LD_LIBRARY_PATH patches to load_ufsda_modules.sh (NOAA-EMC#3236)
  Adding a gefs_arch task to GEFS workflow (NOAA-EMC#3211)
  Add additional GEFS variables needed for AI/ML applications  (NOAA-EMC#3221)
  Add bmat task dependency to marine LETKF task (NOAA-EMC#3224)
  Resolve bug with LMOD_TMOD_FIND_FIRST setting affecting build on WCOSS2 (NOAA-EMC#3229)
  Reinstate product groups (NOAA-EMC#3208)
  Additional fixes for downstream jobs (NOAA-EMC#3187)
  Turn IAU off during staging job for cold start experiments (NOAA-EMC#3215)
  Update the gdas.cd hash and enable GDASApp to run on WCOSS2 (NOAA-EMC#3220)
  Update upload-artifact to v4 (NOAA-EMC#3216)
  Prevent duplicate case generation in generate_workflows.sh (NOAA-EMC#3217)
  Update g-w to cycle with C1152 ATM (NOAA-EMC#3206)
  Separate use of initial increment/perturbation file from REPLAY/+03 ICs  (NOAA-EMC#3119)
  Update gsi_enkf hash and gsi_ver (NOAA-EMC#3207)
  Remove cpus-per-task from APRUN_OCNANALECEN on WCOSS2 (NOAA-EMC#3212)
  Remove 5WAVH from AWIPS GRIB2 parm files (NOAA-EMC#3146)
  Remove multi-grid wave support (NOAA-EMC#3188)
  Add echgres as a dependency for earc (NOAA-EMC#3202)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants