Update dev/ufs-weather-model with develop 2024-01-16

[JessicaMeixner-NOAA]

Pull Request Summary

Update dev/ufs-weather-model with develop 2024-01-16

Description

Update dev/ufs-weather-model with develop 2024-01-16

Note this changes baselines in ufs-weather-model. This is not anticipated but after a lot of searching could not find the reason. The differences are small and we replicate results after the fact.

To be opened after #1342 is merged.

Issue(s) addressed

• no issue

Commit Message

Update dev/ufs-weather-model with develop 2024-1-16

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? within ufs-weather-model. Update WW3 with 20240116 commit  ufs-community/ufs-weather-model#2550
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
• Have the matrix regression tests been run (if yes, please note HPC and compiler)?
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

[JessicaMeixner-NOAA]

Changes should now be seen as just those in: 3952826

[JessicaMeixner-NOAA]

UFS test logs: https://github.com/ufs-community/ufs-weather-model/blob/176ff22243f8fd8367601d99d3d3ef0d79427292/tests/logs/RegressionTests_hera.log

[jkbk2004]

@MatthewMasarik-NOAA Can you review this pr? This pr has been combined to ufs-community/ufs-weather-model#2387

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA Can you review this pr? This pr has been combined to ufs-community/ufs-weather-model#2387

@jkbk2004 I'm currently reviewing. I had found one difference in the test_changes.list from @JessicaMeixner-NOAA's and was looking into that.

cpld_control_pdlib_p8_gnu failed for me as 'unable to complete comparison'. @JessicaMeixner-NOAA should this just be added to the list of failed tests, or let me know if there is some other way you'd like to proceed?

[jkbk2004]

@MatthewMasarik-NOAA Can you review this pr? This pr has been combined to ufs-community/ufs-weather-model#2387

@jkbk2004 I'm currently reviewing. I had found one difference in the test_changes.list from @JessicaMeixner-NOAA's and was looking into that.

cpld_control_pdlib_p8_gnu failed for me as 'unable to complete comparison'. @JessicaMeixner-NOAA should this just be added to the list of failed tests, or let me know if there is some other way you'd like to proceed?

@MatthewMasarik-NOAA sounds like we need new baseline for the case. If that's only issue, we can start working on this pr.

[MatthewMasarik-NOAA]

Code review
Pass

Testing
Pass

OK given by @jkbk2004 for cpld_control_pdlib_p8_gnu needing a new baseline.
test_changes.list.txt
RegressionTests_hera.log.txt

[DeniseWorthen]

I can't see Matt's run directory (closed permissions), but looking at the run directory in the original logs that Jessica posted above (#1343 (comment)) shows

96: WW3 log written to /scratch1/NCEPDEV/stmp2/Jessica.Meixner/FV3_RT/rt_2439762/cpld_debug_pdlib_p8_intel/log.ww3
144:   aft fcst run output time=         360 FBcount=           3 na=           1
196: forrtl: severe (408): fort: (3): Subscript #1 of the array A has value 0 which is less than the lower bound of 1
196:
196: Image              PC                Routine            Line        Source
196: fv3.exe            0000000012225B5F  Unknown               Unknown  Unknown
196: fv3.exe            0000000002B8ED96  w3src4md_mp_w3sds        2256  w3src4md.F90
196: fv3.exe            0000000003032A7C  w3srcemd_mp_w3src        1267  w3srcemd.F90
196: fv3.exe            00000000026390DC  w3wavemd_mp_w3wav        2233  w3wavemd.F90
196: fv3.exe            0000000001B0619A  wav_comp_nuopc_mp        1247  wav_comp_nuopc.F90

[JessicaMeixner-NOAA]

@DeniseWorthen - perhaps a silly question - but I literally don't see where you saw that error

I've submitting a new round of jobs on hercules and hera now from https://github.com/JessicaMeixner-NOAA/WW3/tree/devufs20241211develop20240116

[DeniseWorthen]

@JessicaMeixner-NOAA Thanks for checking....about 5 days ago, you posted a log for UWM RTs earlier in this thread.

EDIT: emacs /scratch1/NCEPDEV/climate/Jessica.Meixner/PR_WW3/UFSPRww3Jan16/ufs-weather-model/tests/logs/log_hera/run_cpld_debug_pdlib_p8_intel.log

[JessicaMeixner-NOAA]

@JessicaMeixner-NOAA Thanks for checking....about 5 days ago, you posted a log for UWM RTs earlier in this thread.

EDIT: emacs /scratch1/NCEPDEV/climate/Jessica.Meixner/PR_WW3/UFSPRww3Jan16/ufs-weather-model/tests/logs/log_hera/run_cpld_debug_pdlib_p8_intel.log

I'm missing something obvious, i posted a log: https://github.com/ufs-community/ufs-weather-model/blob/176ff22243f8fd8367601d99d3d3ef0d79427292/tests/logs/RegressionTests_hera.log and when i got to https://github.com/ufs-community/ufs-weather-model/blob/176ff22243f8fd8367601d99d3d3ef0d79427292/tests/logs/RegressionTests_hera.log i do not see:

96: WW3 log written to /scratch1/NCEPDEV/stmp2/Jessica.Meixner/FV3_RT/rt_2439762/cpld_debug_pdlib_p8_intel/log.ww3
144: aft fcst run output time= 360 FBcount= 3 na= 1
196: forrtl: severe (408): fort: (3): Subscript #1 of the array A has value 0 which is less than the lower bound of 1
196:
196: Image PC Routine Line Source
196: fv3.exe 0000000012225B5F Unknown Unknown Unknown
196: fv3.exe 0000000002B8ED96 w3src4md_mp_w3sds 2256 w3src4md.F90
196: fv3.exe 0000000003032A7C w3srcemd_mp_w3src 1267 w3srcemd.F90
196: fv3.exe 00000000026390DC w3wavemd_mp_w3wav 2233 w3wavemd.F90
196: fv3.exe 0000000001B0619A wav_comp_nuopc_mp 1247 wav_comp_nuopc.F90

so I'm trying to figure out how to get there so I don't miss it again?

[DeniseWorthen]

Look at line 468 in the first log you referenced (EDIT---the first "RegressionTests_hera.log")

* TEST cpld_debug_pdlib_p8_intel: FAILED: RUN DID NOT COMPLETE
-- LOG: /scratch1/NCEPDEV/climate/Jessica.Meixner/PR_WW3/UFSPRww3Jan16/ufs-weather-model/tests/logs/log_hera/run_cpld_debug_pdlib_p8_intel.log

Then look in that run log.

[JessicaMeixner-NOAA]

Look at line 468 in the first log you referenced (EDIT---the first "RegressionTests_hera.log")

* TEST cpld_debug_pdlib_p8_intel: FAILED: RUN DID NOT COMPLETE
-- LOG: /scratch1/NCEPDEV/climate/Jessica.Meixner/PR_WW3/UFSPRww3Jan16/ufs-weather-model/tests/logs/log_hera/run_cpld_debug_pdlib_p8_intel.log

Then look in that run log.

Thank you @DeniseWorthen

[MatthewMasarik-NOAA]

err log from RT: cpld_debug_pdlib_p8_intel

cpld_debug_pdlib_p8_intel.err.txt