Multiple nightly test faillures with GEOSgcm

[mathomp4]

With the merge of the new physics, et al, from @wmputman in GEOSgcm, the nightly tests are throwing multiple failures:

• MOM6 runs fail due to THIM in History. → I know @sdrabenh removed THIM from History in the AGCM. The question is: does the OGCM need it? If not, we remove it from coupled history, if so... 🤷🏼 . Will consult @sanAkel → Draft PR: Fixes for History: Remove THIM and Comment Aerosol Moist Tendencies GEOSgcm_App#424
• OpsL181 runs fail in Extdata → Fixes for WSUB Handling; Make SCM ExtData2G only GEOSgcm_App#425
• SCM runs fail → Numerous issues, see Fixes for SCM Model GEOSgcm_GridComp#725
• GNU no longer passes regression → I thought we solved this with WRITE_RESTART_BY_OSERVER is broken for 4d restarts MAPL#1923 and this GCM definitely has that fix... → It's layout related: GEOSgcm + GNU does not layout regress #541
• GNU MAPL3 Debug does not run → GNU debug and GNU MAPL develop run → They ran but were wrong. I think I fixed this with a push

[mathomp4]

Update on MOM6. It is not just THIM. If I remove THIM from the HISTORY.rc, we now get:

pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DDU2gDT not found>
pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DSS2gDT not found>
pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DOC2gDT not found>
pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DBC2gDT not found>
pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DSU2gDT not found>
pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DNI2gDT not found>
pe=00003 FAIL at line=04893    MAPL_HistoryGridComp.F90                 <Field DBRC2gDT not found>

These fields were commented out in the AGCM history in GEOS-ESM/GEOSgcm_App@708ec45 by @wmputman

We could comment them out. If we do so, we should also do so in every file that has these active:

• GCMPROG.rc.tmpl
• HISTORY.AOGCM-MOM5.rc.tmpl
• HISTORY.AOGCM.rc.tmpl
• HISTORY.BENCHMARK.rc.tmpl
• HISTORY_GEOSIT.rc.tmpl
• HISTORY_MERRA2-DD.rc.tmpl
• HISTORY_S2S.rc.tmpl

THIM is also on in:

• HISTORY.AOGCM-MOM5.rc.tmpl
• HISTORY.BENCHMARK.rc.tmpl
• HISTORY_S2S.rc.tmpl
• HISTORY_MERRA2-DD.rc.tmpl
• HISTORY.AOGCM.rc.tmpl

Good news is, commenting out THIM and the SV fields in History lets MOM6 run.

→ Draft PR in: GEOS-ESM/GEOSgcm_App#424

[mathomp4]

The ExtData issue can be solved by GEOS-ESM/GEOSgcm_App#425