Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update atm var and ens ctests to new GFS COM directory structure (#455) #456

Merged
merged 1 commit into from
May 4, 2023
Merged

update atm var and ens ctests to new GFS COM directory structure (#455) #456

merged 1 commit into from
May 4, 2023

Conversation

RussTreadon-NOAA
Copy link
Contributor

The change in the GFS COM directory structure (g-w PR #1421) requires updates to the ATM variational and local ensemble DA ctests. This PR updates the j-job drivers for the var and ens initialize jobs. Two config files are also updated to be more consistent with g-w.

The data directories in GDASAPP_TESTDATA also need to be remapped to the new GFS COM directory structure. GDASAPP_TESTDATA=/scratch1/NCEPDEV/stmp2/Russ.Treadon/CI/GDASApp/data/lowres/ contains the remapped directories used by the ATM var and ens ctests. The updated directories need to be copied to the official locations on Hera and Orion as part of this PR.

Fixes #455

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this May 1, 2023
@RussTreadon-NOAA
Copy link
Contributor Author

Orion test

Install g-w develop at 608ff65 with GDASApp branch RussTreadon-NOAA:feature/gfscom at 1313ffd.

Set GDASAPP_TESTDATA=/work2/noaa/da/rtreadon/CI/GDASApp/data with 20210323 enkfgdas and gdas directories remapped to the new GFS COM directory structure.

Execute all 50 GDASApp ctests. 44 out of 50 tests pass. Notably, all test_gdasapp_atm_jjob_var and test_gdasapp_atm_jjob_ens jobs pass. The following six soca tests failed:

        1445 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1446 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1447 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
        1448 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)

Check JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP.out. This job failed with the following error:

Traceback (most recent call last):
  File "/work/noaa/da/rtreadon/git/global_workflow/work/sorc/gdas.cd/../../sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py", line 238, in <module>
    envconfig = ufsda.misc_utils.get_env_config(component='soca')
  File "/work/noaa/da/rtreadon/git/global_workflow/work/sorc/gdas.cd/ush/ufsda/misc_utils.py", line 135, in get_env_config
    'OBS_DIR': os.environ['COMOUT'],
  File "/work2/noaa/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'COMOUT'
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[45]: status=1

Hera test
As logged in issue #455 all 6 test_gdasapp_atm_jjob ctests passed yesterday (5/1) on Hera. Will reinstall the current head of g-w develop along with RussTreadon-NOAA:feature/gfscom and run all 50 ctests once Hera is returned to developers after it's monthly maintenance.

@guillaumevernieres
Copy link
Contributor

@RussTreadon-NOAA , the "fix" for the soca tests is (supposed to be) in PR#457.

@RussTreadon-NOAA
Copy link
Contributor Author

Hera test

Install g-w develop at 1242952 with GDASApp branch RussTreadon-NOAA:feature/gfscom at 1313ffd.

Set GDASAPP_TESTDATA=/scratch1/NCEPDEV/stmp2/Russ.Treadon/CI/GDASApp/data with 20210323 enkfgdas and gdas directories remapped to the new GFS COM directory structure.

Execute all 50 GDASApp ctests. As with Orion, 44 out of 50 tests pass on Hera. Notably, all test_gdasapp_atm_jjob_var and test_gdasapp_atm_jjob_ens jobs pass. The following six soca tests failed:

        1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1451 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
        1452 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1453 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)

GDASApp PR #457 addresses these failures.

guillaumevernieres
guillaumevernieres approved these changes May 2, 2023
View reviewed changes
Copy link
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@RussTreadon-NOAA
Copy link
Contributor Author

Thank you @guillaumevernieres for the approval.

@CoryMartin-NOAA , when we merge this PR into develop we need to remap directories in the central GDASAPP_TESTDATA lowres directory. I did this in my copy of GDASAPP_TESTDATA on Hera and Orion. The remapped directories are listed below:

  • Hera: /scratch1/NCEPDEV/stmp2/Russ.Treadon/CI/GDASApp/data/lowres
  • Orion: /scratch1/NCEPDEV/stmp2/Russ.Treadon/CI/GDASApp/data/lowres

When you have time would you please copy my lowres directories to the central GDASAPP_TESTDATA under your id? Once this is done we can merge this PR into develop.

@CoryMartin-NOAA
Copy link
Contributor

@RussTreadon-NOAA Hera is copied, your Orion path looks a little 'Hera-y' :-) If you mean /work2/noaa/da/rtreadon/CI/GDASApp/data/lowres then I've copied those over now too. Do you wish to test before merging now with the correct files in place?

@RussTreadon-NOAA
Copy link
Contributor Author

@RussTreadon-NOAA Hera is copied, your Orion path looks a little 'Hera-y' :-) If you mean /work2/noaa/da/rtreadon/CI/GDASApp/data/lowres then I've copied those over now too. Do you wish to test before merging now with the correct files in place?

Thank you for catching my Orion path error. You got the right path. Yes, let me test before we merge.

@RussTreadon-NOAA
Copy link
Contributor Author

Manually execute ctest -R test_gdasapp on Hera and Orion using GDASAPP_TESTDATA as set by modulefile. On both machines the atm var and ens ctests Passed. On both machines the following tests Failed

The following tests FAILED:
        1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1451 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
        1452 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1453 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1455 - test_gdasapp_land_create_ens (Failed)
        1456 - test_gdasapp_land_imsproc (Failed)
        1457 - test_gdasapp_land_apply_jediincr (Failed)
        1458 - test_gdasapp_land_letkfoi_snowda (Failed)

The soca failures are not unexpected. The land failures may be related to g-w GFS COM updates for land which require corresponding GDASApp land updates.

Will now apply Hera and Orion GW-RT labels to see how automated tests perform.

@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels May 4, 2023
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels May 4, 2023
@emcbot
Copy link

emcbot commented May 4, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Thu May  4 13:33:12 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu May  4 14:32:07 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu May  4 14:44:49 UTC 2023
Tests: 80% tests passed, 10 tests failed out of 50
	1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1451 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
	1452 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1453 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1455 - test_gdasapp_land_create_ens (Failed)
	1456 - test_gdasapp_land_imsproc (Failed)
	1457 - test_gdasapp_land_apply_jediincr (Failed)
	1458 - test_gdasapp_land_letkfoi_snowda (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/456/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels May 4, 2023
@emcbot
Copy link

emcbot commented May 4, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Thu May  4 08:32:21 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu May  4 09:41:05 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu May  4 09:51:26 CDT 2023
Tests: 80% tests passed, 10 tests failed out of 50
	1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1451 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
	1452 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1453 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1455 - test_gdasapp_land_create_ens (Failed)
	1456 - test_gdasapp_land_imsproc (Failed)
	1457 - test_gdasapp_land_apply_jediincr (Failed)
	1458 - test_gdasapp_land_letkfoi_snowda (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/456/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels May 4, 2023
@CoryMartin-NOAA
Copy link
Contributor

@jiaruidong2017 I think the GDASApp tests need updated to use the new COM structure too... I'm guessing that is why those are failing now

@RussTreadon-NOAA
Copy link
Contributor Author

@CoryMartin-NOAA , @guillaumevernieres , @jiaruidong2017 : Automated GW-RT tests fail in the same manner as manual tests. These failures are not unexpected. The atm var and ens tests pass. These tests are the focus of this PR. Given this, I think we can merge this PR into develop.

If no one objects, I'll merge no later than noon today (5/4/2023).

@RussTreadon-NOAA RussTreadon-NOAA merged commit f5ae69c into NOAA-EMC:develop May 4, 2023
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/gfscom branch May 4, 2023 17:15
@jiaruidong2017
Copy link
Collaborator

Sorry for the late reply, I just notice this, because I couldn't receive any notifications email from GitHub. Please send me an email if you have the important information. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guillaumevernieres guillaumevernieres guillaumevernieres approved these changes

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels
hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update paths for UFS-DA var and ens ctests
5 participants
@RussTreadon-NOAA @guillaumevernieres @CoryMartin-NOAA @emcbot @jiaruidong2017