Add New Coupling Option of LND (Noah-MP) and ATM (FV3) (#171)

* update hash of ufs-weather-model

* add atm-land compile option

* fix camke_setting issue

* change if condition

* add initial templates for ATML

* make use common templates

* merge model_configure templates

* fix missing change

* initial setup script

* update setup script

* fix fill_jinja issue on setup script

* change launch script to template

* fix missing parms in templates

* make hpc parm calculated in script

* fix missing change

* change name ncores

* add missinog max cores

* fix format issue in ufs.configure

* add confgi.atml.yaml

* update script to add atml option

* remove tstub and tpath

* remove atm_model from config

* fix mismatch in script

* fix coldstart issue on forecast

* add ufs_utils for cold start IC

* update module files

* add fcst_ic task

* change template for chgres_cube

* fix namelist issue on chgres_cube

* fix missing filed_table in forecast

* change hash of ufs_utils

* update atml scripts

* change file name and err_exit

* add missing change

* update scripts

* fix ctest failure

* fix bugs in forecast script

* change file path for forecast we2e

* add memory to fcst_ics

* update pr template

* add ioda convert capability to prep_obs

* fix typos

* modify ioda conv script to match time stamp with jedi

* add detect_platform to setup script

* set time adjustment flexible

* fix missing global attributes in ioda conv script

* fix LND coldstart issues

* fix ctest failure

* update forecast script for atml

* fix restart file name issue

* add obs plot

* update restart plot part

* turn off checksum for restart of ufs_model

* add atml warm start sample

* add python script to plot sfc_data

* fix python issues on hera

* move jedi_py_ver

* increase wtime of prep_obs for orion

.github/pull_request_template.md

@@ -9,6 +9,7 @@ Provide a detailed description of what this PR does. What bug does it fix, or wh
 ## Subcomponents involved:
 - [ ] apply_incr.fd (NOAA-PSL/land-apply_jedi_incr)
 - [ ] ufs_model.fd (ufs-community/ufs-weather-model)
+- [ ] UFS_UTILS.fd (ufs-community/UFS_UTILS)
 - [ ] none
 
 ## Linked PR's and Issues:

.gitignore

@@ -6,14 +6,8 @@ sorc/conda/
 sorc/lib/
 sorc/lib64
 parm/conda_loc
-parm/*.xml
-parm/*.db
-parm/*.out
-parm/*.err
-parm/*.log
-parm/parm_xml.yaml
-parm/land_analysis.yaml
-parm/log.*
+parm/config.yaml
+ush/__pycache__/
 
 __pycache__
 *.swp

.gitmodules

@@ -2,8 +2,13 @@
 	path = sorc/ufs_model.fd
 	url = https://github.com/ufs-community/ufs-weather-model.git
 	ignore = dirty
+
 [submodule "sorc/apply_incr.fd"]
 	path = sorc/apply_incr.fd
 	url = https://github.com/NOAA-PSL/land-apply_jedi_incr.git
         ignore = dirty
 
+[submodule "sorc/UFS_UTILS.fd"]
+	path = sorc/UFS_UTILS.fd
+	url = https://github.com/ufs-community/UFS_UTILS.git
+	ignore = dirty

README.md

@@ -1,12 +1,12 @@
-# UFS Offline Land Data Assimilation System
+# UFS Land Data Assimilation System
 
 The Unified Forecast System (UFS) is a community-based, coupled, comprehensive Earth modeling system. It is designed to be the source system for NOAA's operational numerical weather prediction applications while enabling research, development, and contribution opportunities for the broader Weather Enterprise. For more information about the UFS, visit the UFS Portal at https://ufs.epic.noaa.gov/.
 
-The UFS includes [multiple applications](https://ufs.epic.noaa.gov/applications/) that support different forecast durations and spatial domains. This repository hosts the source code for the UFS Land Data Assimilation (DA) System. Land DA is an offline version of the Noah Multi-Physics (Noah-MP) land surface model (LSM) used in the UFS Weather Model (WM). Its data assimilation framework uses the Joint Effort for Data assimilation Integration (JEDI) software stack, which includes the Object-Oriented Prediction System (OOPS) for the data assimilation algorithm, the Interface for Observation Data Access (IODA) for observation formatting and processing, and the Unified Forward Operator (UFO) for comparing model forecasts and observations. 
+The UFS includes [multiple applications](https://ufs.epic.noaa.gov/applications/) that support different forecast durations and spatial domains. This repository hosts the source code for the UFS Land Data Assimilation (DA) System. Land DA applies the Noah Multi-Physics (Noah-MP) land surface model (LSM) of the UFS Weather Model (WM) as the key component. Its data assimilation framework uses the Joint Effort for Data assimilation Integration (JEDI) software stack, which includes the Object-Oriented Prediction System (OOPS) for the data assimilation algorithm, the Interface for Observation Data Access (IODA) for observation formatting and processing, and the Unified Forward Operator (UFO) for comparing model forecasts and observations. 
 
-The offline Noah-MP LSM is a standalone, uncoupled model used to execute land surface simulations. In this traditional uncoupled mode, near-surface atmospheric forcing data is required as input forcing. This LSM simulates soil moisture (both liquid and frozen), soil temperature, skin temperature, snow depth, snow water equivalent (SWE), snow density, canopy water content, and the energy flux and water flux terms of the surface energy balance and surface water balance. Its data assimilation framework applies the Local Ensemble Transform Kalman Filter-Optimal Interpolation (LETKF-OI) algorithm to combine the state-dependent background error derived from an ensemble forecast with the observations and their corresponding uncertainties to produce an analysis ensemble (Hunt et al., 2007).
+The Noah-MP LSM is used to execute land surface simulations. A near-surface atmospheric forcing data is required as input forcing. This LSM simulates soil moisture (both liquid and frozen), soil temperature, skin temperature, snow depth, snow water equivalent (SWE), snow density, canopy water content, and the energy flux and water flux terms of the surface energy balance and surface water balance. Its data assimilation framework applies the Local Ensemble Transform Kalman Filter-Optimal Interpolation (LETKF-OI) algorithm to combine the state-dependent background error derived from an ensemble forecast with the observations and their corresponding uncertainties to produce an analysis ensemble (Hunt et al., 2007).
 
-The Noah-MP LSM has evolved through community efforts to pursue and refine a modern-era LSM suitable for use in the National Centers for Environmental Prediction (NCEP) operational weather and climate prediction models. This collaborative effort continues with participation from entities such as NCAR, NCEP, NASA, and university groups. The development branch of the Land DA System is continually evolving as the system undergoes open development. The latest Land DA release (v1.2.0) represents a snapshot of this continuously evolving system. 
+The Noah-MP LSM has evolved through community efforts to pursue and refine a modern-era LSM suitable for use in the National Centers for Environmental Prediction (NCEP) operational weather and climate prediction models. This collaborative effort continues with participation from entities such as NCAR, NCEP, NASA, and university groups. The development branch of the Land DA System is continually evolving as the system undergoes open development. The latest Land DA release (v2.0.0) represents a snapshot of this continuously evolving system. 
 
 The Land DA System User's Guide associated with the development branch is at: https://land-da-workflow.readthedocs.io/en/develop/, while the guide specific to the Land DA v2.0.0 release can be found at: https://land-da-workflow.readthedocs.io/en/release-public-v2.0.0/. Users may download data for use with the most recent release from the [Land DA data bucket](https://registry.opendata.aws/noaa-ufs-land-da/). The [Land DA Docker Hub](https://hub.docker.com/r/noaaepic/ubuntu22.04-intel21.10-landda) hosts Land DA containers. These containers package the Land DA System together with all its software dependencies for an easier experience building and running Land DA.
 

jobs/JLANDDA_ANALYSIS

@@ -73,15 +73,14 @@ fi
 mkdir -p ${COMOUT}
 export COMOUThofx="${COMOUThofx:-${COMOUT}/hofx}"
 mkdir -p ${COMOUThofx}
+export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
+mkdir -p ${COMOUTplot}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
+export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
+mkdir -p ${DATA_RESTART}
 export DATA_HOFX="${DATA_HOFX:-${DATAROOT}/DATA_SHARE/hofx}"
 mkdir -p ${DATA_HOFX}
-
-# Set other dates
-export PTIME=$($NDATE -24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_FCST_IC

@@ -0,0 +1,105 @@
+#!/bin/bash
+
+date
+export PS4='+ $SECONDS + '
+set -xue
+#
+#-----------------------------------------------------------------------
+#
+# Set the NCO standard environment variables (Table 1, pp.4)
+#
+#-----------------------------------------------------------------------
+#
+export USHlandda="${HOMElandda}/ush"
+export EXEClandda="${HOMElandda}/exec"
+export PARMlandda="${HOMElandda}/parm"
+export FIXlandda="${HOMElandda}/fix"
+export SCRIPTSlandda="${HOMElandda}/scripts"
+#
+#-----------------------------------------------------------------------
+#
+# Define job and jobid by default for rocoto
+#
+#-----------------------------------------------------------------------
+#
+WORKFLOW_MANAGER="${WORKFLOW_MANAGER:-rocoto}"
+if [ "${WORKFLOW_MANAGER}" = "rocoto" ]; then
+  if [ "${SCHED}" = "slurm" ]; then
+    job=${SLURM_JOB_NAME}
+    pid=${SLURM_JOB_ID}
+  elif [ "${SCHED}" = "pbspro" ]; then
+    job=${PBS_JOBNAME}
+    pid=${PBS_JOBID}
+  else
+    job="task"
+    pid=$$
+  fi
+  jobid="${job}.${PDY}${cyc}.${pid}"
+fi
+#
+#-----------------------------------------------------------------------
+#
+# Create a temp working directory (DATA) and cd into it.
+#
+#-----------------------------------------------------------------------
+#
+export DATA="${DATA:-${DATAROOT}/${jobid}}"
+mkdir -p $DATA
+cd $DATA
+#
+#-----------------------------------------------------------------------
+#
+# Define NCO environment variables and set COM type definitions.
+#
+#-----------------------------------------------------------------------
+#
+export NET="${NET:-landda}"
+export RUN="${RUN:-landda}"
+
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
+[[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
+if [ "${MACHINE}" = "WCOSS2" ]; then
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMINgdas="${COMINgdas:-$(compath.py ${envir}/gdas/${gdas_ver})}"
+  export COMINgfs="${COMINgfs:-$(compath.py ${envir}/gfs/${gfs_ver})}"
+else
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMINgdas="${COMINgdas:-${FIXlandda}/DATA_gdas}"
+  export COMINgfs="${COMINgfs:-${FIXlandda}/DATA_gfs}"
+fi
+
+mkdir -p ${COMOUT}
+
+#
+#-----------------------------------------------------------------------
+#
+# Call the ex-script for this J-job.
+#
+#-----------------------------------------------------------------------
+#
+export pgmout="${DATA}/OUTPUT.$$"
+env
+
+${SCRIPTSlandda}/exlandda_fcst_ic.sh
+export err=$?; err_chk
+
+if [ -e "$pgmout" ]; then
+  cat $pgmout
+fi
+#
+#-----------------------------------------------------------------------
+#
+# Whether or not working directory DATA should be kept.
+#
+#-----------------------------------------------------------------------
+#
+if [ "${KEEPDATA}" = "NO" ]; then
+  rm -rf ${DATA}
+fi
+date

jobs/JLANDDA_FORECAST

@@ -74,14 +74,16 @@ fi
 
 mkdir -p ${COMOUT}
 
+# RESTART directory in COMOUT
+mkdir -p ${COMOUT}/RESTART
+# plot directory in COMOUT
+export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
+mkdir -p ${COMOUTplot}
+
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
 mkdir -p ${DATA_RESTART}
 
-# Set other dates
-export NTIME=$($NDATE 24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PLOT_STATS

@@ -75,13 +75,8 @@ export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
 mkdir -p ${COMOUTplot}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_HOFX="${DATA_HOFX:-${DATAROOT}/DATA_SHARE/hofx}"
 mkdir -p ${DATA_HOFX}
-
-# Set other dates
-export NTIME=$($NDATE 24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_POST_ANAL

@@ -73,13 +73,9 @@ fi
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
 mkdir -p ${DATA_RESTART}
 
-# Set other dates
-export NTIME=$($NDATE 24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PREP_OBS

@@ -73,13 +73,9 @@ fi
 mkdir -p ${COMOUT}
 export COMOUTobs=${COMOUTobs:-${COMOUT}/obs}
 mkdir -p ${COMOUTobs}
+export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
+mkdir -p ${COMOUTplot}
 
-# Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
-
-# Set other dates
-export PTIME=$($NDATE -24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PRE_ANAL

@@ -73,13 +73,9 @@ fi
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
 mkdir -p ${DATA_RESTART}
 
-# Set other dates
-export PTIME=$($NDATE -24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

modulefiles/build_container_intel.lua

@@ -71,5 +71,5 @@ setenv("CC", "mpiicc")
 setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 
-setenv("JEDI_INSTALL", pathJoin(os.getenv("EPICHOME"),""))
+setenv("JEDI_PATH", pathJoin(os.getenv("EPICHOME"),""))
 

modulefiles/build_hera_intel.lua

@@ -32,6 +32,12 @@ load(pathJoin("w3emc", w3emc_ver))
 load(pathJoin("gftl-shared", gftl_shared_ver))
 load(pathJoin("mapl", mapl_ver))
 load(pathJoin("scotch", scotch_ver))
+
+load(pathJoin("nemsio", nemsio_ver))
+load(pathJoin("sfcio", sfcio_ver))
+load(pathJoin("sigio", sigio_ver))
+load(pathJoin("nccmp", nccmp_ver))
+
 load(pathJoin("prod_util", prod_util_ver))
 load(pathJoin("ufs-pyenv", ufs_pyenv_ver))
 
@@ -40,5 +46,5 @@ setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 setenv("CMAKE_Platform", "hera.intel")
 
-setenv("EPICHOME", "/scratch2/NAGAPE/epic/UFS_Land-DA_Dev")
-setenv("JEDI_INSTALL", "/scratch2/NAGAPE/epic/UFS_Land-DA_Dev/jedi_v7")
+setenv("EPICHOME", "/scratch2/NAGAPE/epic/UFS_Land-DA_v2.1")
+setenv("JEDI_PATH", "/scratch2/NAGAPE/epic/UFS_Land-DA_v2.1/jedi_v7_ic")

modulefiles/build_hercules_intel.lua

@@ -32,6 +32,12 @@ load(pathJoin("w3emc", w3emc_ver))
 load(pathJoin("gftl-shared", gftl_shared_ver))
 load(pathJoin("mapl", mapl_ver))
 load(pathJoin("scotch", scotch_ver))
+
+load(pathJoin("nemsio", nemsio_ver))
+load(pathJoin("sfcio", sfcio_ver))
+load(pathJoin("sigio", sigio_ver))
+load(pathJoin("nccmp", nccmp_ver))
+
 load(pathJoin("prod_util", prod_util_ver))
 load(pathJoin("ufs-pyenv", ufs_pyenv_ver))
 
@@ -43,5 +49,5 @@ setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 setenv("CMAKE_Platform", "hercules.intel")
 
-setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_Dev")
-setenv("JEDI_INSTALL", "/work/noaa/epic/UFS_Land-DA_Dev/jedi_v7_hercules")
+setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_v2.1")
+setenv("JEDI_PATH", "/work/noaa/epic/UFS_Land-DA_v2.1/jedi_v7_ic_hercules")

modulefiles/build_orion_intel.lua

@@ -32,6 +32,12 @@ load(pathJoin("w3emc", w3emc_ver))
 load(pathJoin("gftl-shared", gftl_shared_ver))
 load(pathJoin("mapl", mapl_ver))
 load(pathJoin("scotch", scotch_ver))
+
+load(pathJoin("nemsio", nemsio_ver))
+load(pathJoin("sfcio", sfcio_ver))
+load(pathJoin("sigio", sigio_ver))
+load(pathJoin("nccmp", nccmp_ver))
+
 load(pathJoin("prod_util", prod_util_ver))
 load(pathJoin("ufs-pyenv", ufs_pyenv_ver))
 
@@ -43,5 +49,5 @@ setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 setenv("CMAKE_Platform", "orion.intel")
 
-setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_Dev")
-setenv("JEDI_INSTALL", "/work/noaa/epic/UFS_Land-DA_Dev/jedi_v7_stack1.6")
+setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_v2.1")
+setenv("JEDI_PATH", "/work/noaa/epic/UFS_Land-DA_v2.1/jedi_v7_ic_orion")

modulefiles/build_singularity_intel.lua

@@ -100,6 +100,6 @@ setenv("CC", "mpiicc")
 setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 
-setenv("JEDI_INSTALL", pathJoin(os.getenv("EPICHOME"),""))
+setenv("JEDI_PATH", pathJoin(os.getenv("EPICHOME"),""))
 
 whatis("Description: UFS build environment")

modulefiles/tasks/hera/task.analysis.lua

@@ -1,13 +1,16 @@
-prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
+prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_unienv"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
 load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))
-
+load(pathJoin("py-cartopy", py_cartopy_ver))
 load(pathJoin("py-jinja2", py_jinja2_ver))
+load(pathJoin("py-matplotlib", py_matplotlib_ver))
 load(pathJoin("py-netcdf4", py_netcdf4_ver))
 load(pathJoin("py-numpy", py_numpy_ver))
 load(pathJoin("py-pyyaml", py_pyyaml_ver))
+load(pathJoin("py-scipy", py_scipy_ver))
+load(pathJoin("py-xarray", py_xarray_ver))
 

modulefiles/tasks/hera/task.fcst_ic.lua

@@ -0,0 +1,20 @@
+prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
+
+load(pathJoin("stack-intel", stack_intel_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
+load(pathJoin("stack-python", stack_python_ver))
+
+load(pathJoin("jasper", jasper_ver))
+load(pathJoin("hdf5", hdf5_ver))
+load(pathJoin("netcdf-c", netcdf_c_ver))
+load(pathJoin("netcdf-fortran", netcdf_fortran_ver))
+load(pathJoin("parallelio", parallelio_ver))
+load(pathJoin("esmf", esmf_ver))
+load(pathJoin("g2", g2_ver))
+load(pathJoin("prod_util", prod_util_ver))
+
+load(pathJoin("py-jinja2", py_jinja2_ver))
+load(pathJoin("py-netcdf4", py_netcdf4_ver))
+load(pathJoin("py-numpy", py_numpy_ver))
+load(pathJoin("py-pyyaml", py_pyyaml_ver))
+

modulefiles/tasks/hera/task.prep_obs.lua

@@ -1,7 +1,15 @@
-prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
+prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_unienv"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
 load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))
+
+load(pathJoin("py-cartopy", py_cartopy_ver))
+load(pathJoin("py-matplotlib", py_matplotlib_ver))
+load(pathJoin("py-netcdf4", py_netcdf4_ver))
+load(pathJoin("py-numpy", py_numpy_ver))
+load(pathJoin("py-pandas", py_pandas_ver))
+load(pathJoin("py-pyyaml", py_pyyaml_ver))
+