Add fetch job and update stage_ic to work with fetched ICs

[DavidGrumm-NOAA]

Description

Most jobs require the initial conditions to be available on local disk. The existing “stage_ic” task copies/stages these initial condition into the experiment's COM directory. This PR for the “fetch” task extends that functionality to copy from HPSS (on HPSS-accessible machines) into COM.

Resolves #2988

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? YES
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• In process of being tested on Hera

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• [] I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidGrumm-NOAA]

I am in the process of testing.

[DavidGrumm-NOAA]

To test my code, I ran create_experiment with the short yaml C48_ATM.yaml, (which created /scratch1/NCEPDEV/global/David.Grumm/G_WF_2988/testroot_1/EXPDIR and COMROOT) by :

HPC_ACCOUNT="fv3-cpu" MY_TESTROOT="/scratch1/NCEPDEV/global/David.Grumm/G_WF_2988/testroot_1" RUNTESTS=${MY_TESTROOT} pslot="1306a_2988" ./create_experiment.py --yaml ../ci/cases/pr/C48_ATM.yaml

… which completed without error or warning messages.

From within that EXPDIR I ran rocotorun:
rocotorun -w ./1306a_2988/1306a_2988.xml -d ./1306a_2988/1306a_2988.db

… which completed without error or warning messages. There was also no output to stdout, which I did not expect as I had placed a few diagnostic prints in my code. I verified that I am my current branch.

Runniing rocotostat gives me:

CYCLE TASK JOBID STATE EXIT STATUS TRIES DURATION

========================================================================
202103231200 gfs_stage_ic druby://10.184.8.62:37937 SUBMITTING - 0 0.0
202103231200 gfs_fcst_seg0 - - - - -
202103231200 gfs_atmos_prod_f000 - - - - -
etc.
… and this appears unchanged (at least for the 4 hours since I ran rocotorun)

I have 2 questions:

• Am I not running my code, as the diagnostic prints do not appear ? ( I was able to run similarly modified versions of exglobal_fetch.py and fetch.py earlier when I had run fetch.sh, but I was running that script from the same directory. It simply may be a matter of manually resetting my PATH)

• Rocotostat indicates that a job has been submitted (presumably with another version of the code)
  Shouldn’t I expect it to progress ?

[DavidHuber-NOAA]

Rocoto is not a fully automated system. For each invocation of rocotorun, it checks the status of running jobs and updates its database accordingly, checks to see if any new jobs have met their prerequisites, and submits jobs to the queue that are ready to run. rocotostat simply reads the database and reports its contents. Thus, rocotorun must be run every few minutes to continuously submit jobs. Conveniently, in your EXPDIR, there is a file with extension .crontab. Copy the contents of this file to your crontab (which you can edit via the crontab -e command). Now your experiment should run continuously. For more details, read the docs.

[DavidHuber-NOAA]

Just some whitespace cleanup (most of it mine).

[DavidGrumm-NOAA]

I updated the crontab.

[DavidGrumm-NOAA]

Removed extraneous white space from fetch.py and recommitted; still testing.

[DavidGrumm-NOAA]

I moved the fetch options to be in the run_options dict.

[DavidGrumm-NOAA]

Ok. I will revert those back. Also- Walter: For some reason I don't see your comment on the PR page (only in the email)

…

On Thu, Jan 23, 2025 at 1:44 PM Walter Kolczynski - NOAA < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In workflow/hosts/gaeac5.yaml <#3141 (comment)> : > @@ -24,5 +24,7 @@ LOCALARCH: 'NO' ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}' MAKE_NSSTBUFR: 'NO' MAKE_ACFTBUFR: 'NO' -SUPPORTED_RESOLUTIONS: ['C384', 'C192', 'C96', 'C48'] +DO_FETCH_HPSS: 'NO' +SUPPORTED_RESOLUTIONS: ['C1152', 'C768', 'C384', 'C192', 'C96', 'C48'] I think the higher resolutions need to stay in the supported list. — Reply to this email directly, view it on GitHub <#3141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BKF4TU5RNQ272366FH75ELD2ME2CHAVCNFSM6AAAAABTCZFW46VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNZQG4ZDMNJYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DavidHuber-NOAA]

These host files have extra whitespace in them. Could you please remove that? There should be no difference between these and develop.

[DavidGrumm-NOAA]

Are you referring to (for that file) a new line at the end ? I added that as I've read that UNIX tools expect a new line at the end. I can remove it from that file and others to agree with the develop versions.

[DavidHuber-NOAA]

Yes, the new line character is expected, but most editors put this in automatically. If, after removing these blank lines, we see artifacts indicating that there isn't a new line character at the end of the file, we can diagnose the issue in emacs.

[DavidGrumm-NOAA]

Ok, thanks for the explanation. I will remove them.

[DavidGrumm-NOAA]

Removed extraneous newlines.

[DavidHuber-NOAA]

Try adding this to your .emacs file:

(setq require-final-newline 1)

Then open, edit (add, then delete a character), and save the file. Lastly, run xxd on one of the files and verify that the last hex byte is 0A. This indicates that a new line was written at the end of the file.

[DavidGrumm-NOAA]

I tried it - a nice, useful feature. However, I may not always want there to be a new line at the end. So although it seems that I can turn off the prompt (when saving) asking if I want to append a newline, I may then forget that that would be the default. I will read up on this some more. Thanks.

…

On Fri, Jan 24, 2025 at 4:19 PM David Huber ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ On workflow/hosts/awspw.yaml <#3141 (comment)> : Try adding this to your .emacs file: (setq require-final-newline 1) Then open, edit (add, then delete a character), and save the file. Lastly, run xxd on one of the files and verify that the last hex byte is 0A. This indicates that a new line was written at the end of the file. — Reply to this email directly, view it on GitHub <#3141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BKF4TU63STTAXYG5WL4WB332MKU7LAVCNFSM6AAAAABTCZFW46VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNZTGU4TGMZZGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DavidGrumm-NOAA]

Walter - After reconciling your comments with Dave Huber's, I think I have made all of the agreed-upon changes from your review.

…

On Thu, Jan 23, 2025 at 2:28 PM Walter Kolczynski - NOAA < ***@***.***> wrote: ***@***.**** requested changes on this pull request. ------------------------------ In ci/cases/yamls/gfs_defaults_ci.yaml <#3141 (comment)> : > + FETCHDIR: "/NCEPDEV/emc-global/1year/David.Grumm/test_data" + DO_FETCH_HPSS: "YES" I don't think we want to turn this on in CI tests. ------------------------------ In parm/config/gfs/config.base <#3141 (comment)> : > +# TODO once tarballs are available for CI at other resolutions/start types, etc, remove this if block +if [[ "${APP}" == "ATM" || "${APP}" == "S2SW" ]] && \ + [[ "${EXP_WARM_START}" == ".false." && "${CASE}" == "C48" ]]; then + export ***@***.***_FETCH_HPSS@" # pull from HPSS ## 123124 DG change to YES, or next + export DO_FETCH_LOCAL="NO" # pull from local +fi I know this is temporary anyway, but I don't think it should be handled here at all. ------------------------------ In parm/config/gfs/config.fetch <#3141 (comment)> : > +# Get task specific resources +source "${EXPDIR}/config.resources" fetch To be consistent with other config files, sourcing resources should be first. ------------------------------ In workflow/rocoto/gfs_tasks.py <#3141 (comment)> : > @@ -29,6 +38,27 @@ def stage_ic(self): 'command': f'{self.HOMEgfs}/jobs/rocoto/stage_ic.sh', 'job_name': ***@***.***', 'log': ***@***.***@***@***@***.***/{task_name}.log', + 'maxtries': '&MAXTRIES;', + 'dependency': dependencies + } + + task = rocoto.create_task(task_dict) + + return task + + def fetch(self): Move this task def above stage_ic so they appear in the logical order. ------------------------------ In parm/fetch/gfs_S2SW_cold_forecast-only.yaml.j2 <#3141 (comment)> : > @@ -0,0 +1,37 @@ +{% set cycle_YMDH = current_cycle | to_YMDH %} +{% set cycle_YMD = current_cycle | to_YMD %} +{% set cycle_HH = current_cycle | strftime("%H") %} +{% set p_cycle_YMD = previous_cycle | to_YMD %} Can we please use 'previous_orprev_instead ofp_` for these varnames? — Reply to this email directly, view it on GitHub <#3141 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BKF4TUZE62VLZ3P5AWYVTKD2ME7FDAVCNFSM6AAAAABTCZFW46VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNZQHAYDENBZG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[emcbot]

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3141

Experiment C48_ATM_939ce24d Completed 1 Cycles: *SUCCESS* at Fri Jan 24 15:54:49 CST 2025
Experiment C48mx500_3DVarAOWCDA_939ce24d Completed 2 Cycles: *SUCCESS* at Fri Jan 24 16:12:59 CST 2025
Experiment C48mx500_hybAOWCDA_939ce24d Completed 2 Cycles: *SUCCESS* at Fri Jan 24 16:13:09 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_939ce24d Completed 1 Cycles: *SUCCESS* at Fri Jan 24 16:37:21 CST 2025
Experiment C96C48_hybatmDA_939ce24d Completed 3 Cycles: *SUCCESS* at Fri Jan 24 17:01:39 CST 2025
Experiment C96_atm3DVar_939ce24d Completed 3 Cycles: *SUCCESS* at Fri Jan 24 17:13:42 CST 2025
Experiment C48_S2SW_939ce24d Completed 1 Cycles: *SUCCESS* at Fri Jan 24 17:19:35 CST 2025
Experiment C48_S2SWA_gefs_939ce24d Completed 1 Cycles: *SUCCESS* at Fri Jan 24 18:50:57 CST 2025