UFS-CAT Troubleshooting help

[janahaddad]

This is a space for the UFS-CAT to add their questions and comments to the UFS-Coastal development team.

[janahaddad]

Questions & Answers (+ some discussion) from chat:

So we no longer need to check out branch feature/coastak_app?

• Ufuk: I made that branch default at this point. If you don't provide the branch, it will checkout feature/coastak_app by default

The sea ice model is CICE6?

• Ufuk: Yes, we are using same code base with UFS Weather Model. We did not start to populate CICE configurations coupled with UFS Coastal Specific ocean models but this will be done in near future. Probably starting with ROMS and SCHSIM
• CICE hasn't yet been activated in ufs-coastal. still working on the hydrodynamic model componenets to ensure those have correct/working NUOPC interfaces

Are the Ice and Waves model components different from what's under the ufs-weather?

• Ufuk: Same ice and wave model are used in here, CICE and WW3

Has either ROMS or SCHISM been coupled to CICE previously?

• Ufuk: ROMS has coupled with CICE internally. It is not tested with external ice component. So, we might need to spend some time to create that configuration. I hope their CICE model is not different to much than the one found in UFS Weather Model. I am not sure about SCHSIM but as I know the coupling interface of SCHSIM is not real to couple with CICE yet.
  ready
• Comment from Alex K: Our OSU collaborators are trying to use this whole UFS-Coastal machine right now to couple ROMS-CICE. We are working on a presumption that CICE is part of UFS-Coastal. Interesting to learn it is not.
• Ufuk: CICE is part of UFS Coastal (since it is forked from UFS Weather Model) but ROMS+CICE configuration is not tested yet.
• Alex K: OK, I hope we will be able to test it, working collaboratively. ... We are done testing CICE6 with an ocean data layer, and ready to couple two-way via NUOPC, and gearing up with Bering Sea assessments using SCHISM
• Ufuk: If ROMS and CICE is working standalone fine. Then, we could start to populate the coupled configuration using those cases to see what happens and start to solve the issues that could arise one by one.
• Ayumi: Alex, for Great Lakes, we use FVCOM-CICE. and also testing CICE6 standalone for future coupling with a hydrodynamic model using ufs coastal.
• Saeed: Ayumi -- Thank you for your clarification. Yes, this was my understanding of the situation. ROMS does not have an embedded CICE subroutine.
• AJ: I would say that CICE is a module/component of a model system, like biological and sediment module. I would not say that running ice module in roms and fvcom is a coupling
• Denise: CICE is a component model, running separately but exchanging fields to/from the ocean component through the CICE NUOPC cap.
• Saeed: AJ, ufs-coastal will use model wrapped in caps and couple through ESMF/NUOP. The NOAA plan is to make coupling plug and play for all models as external components and go away from specific hard-coded internal modules. Ufuk and others on the call may be able to provide more context.
• Alex K.: Re: "I would not say that running ice module in roms and fvcom is a coupling". As planned CICE and ROMS (or any other ocean model) will be coupled two-way using NUOPC. CICE and ROMS do exchange information two-way, so this is a classical example of coupling.

How far away is coastal app from operational application? To do dynamic two way coupling, does NCEP need to run weather and wave models under coastal app?

• There is work to do but need more active contribution from the community.
• Saeed: I think, the active contribution and testing is the key. 1st from developers and 2nd from NOAA colleagues and partners to implement the codes into operational systems.
• Greg S.: AJ & Saeed, also upcoming COMT and other projects with a focus for those being UFS. Agree partnership w NWS/EMC is key for two-way coupled systems

[ayumifuji]

Hello UFS Coastal Team!

Two questions regarding running a regression test on Frontera:

1. When input datasets for the regression tests become available on Frontera, can you please let us know and share a path? I tried to run a SCHISM+DATM test on Frontera, but failed as it did not find a input data (input-data-20221101/DATM_ATMMESH_input_data/Ike/*).
2. Relevant to the above, I had to change "CREATE_BASELINE=false" --> "CREATE_BASELINE=true" in rt.sh on Frontera. Is this ok?

Ayumi

[uturuncoglu]

@ayumifuji Sorry for late response. Since I did not subscribe, I did not get this. Anyway, here are my comments to your questions,

1. All the input data are available in /work2/01118/tg803972/frontera/RT/NEMSfv3gfs/input-data-20221101/ including DATM_ATMMESH_input_data directory. The ufs-coastal has this information in rt.sh,
   https://github.com/oceanmodeling/ufs-coastal/blob/bda458a120ca80e707a9baf4bf692a83ea9af34b/tests/rt.sh#L533
   There was recent issue in the system that was preventing to access to /work2. So, you might try in that time by luck. @pvelissariou1 also confirm that RTs are running without any issue. Please try again and if you still have issue, please let us know and we could look at together. There could be also permissions issue.

2. You don't need to set CREATE_BASELINE. The baseline are already created on Frontera but I am not sure they are up-to-date or not. @pvelissariou1 in your test are SCHISM+DATM configuration also pass the baseline comparison? In any case, if you want to create baseline from scratch, you could just pass -c flag to rt.sh.

[ayumifuji]

Thank you, Ufuk!
Re 1: Ah, my bad. I mistakenly used a wrong path for DISKNM. Now it's working!

Re 2: CREASE_BASELINE=Faluse still does not work but adding -c to rt.sh worked. Thank you! @pvelissariou1

[pvelissariou1]

@ayumifuji We will report in this discussion as we are making updates to the test cases.

[ayumifuji]

Hi @uturuncoglu and @pvelissariou1
So I got further in my ./rt.sh -l rt_coastal.conf -a OCE22003 -k -c -n coastal_ike_shinnecock_atm2sch intel.
However, I saw the following error:

• ./run_test.sh /work2/09028/ayumif/frontera/ufs-coastal/ufs-coastal/tests /scratch1/09028/ayumif/frontera/ayumif/FV3_RT/rt_148440 coastal_ike_shinnecock_atm2sch 001 11_intel
  ++ echo 'rt.sh error on line 956'

I see done < $TESTS_FILE at line 956 of rt.sh, but do not know why error happens here.... Any thoughts?

It generated log files under /work2/09028/ayumif/frontera/ufs-coastal/ufs-coastal/tests/logs/log_frontera/.

Thank you!

[pvelissariou1]

@ayumifuji Ayumi most likely a misconfig error. There are log files where errors are reported. The log files are located in tests/logs/log_frontera and also in your /scratch1/09028/ayumif/ stmp folder for your run.
The error is because in your rt.sh file the DISKNM value is wrong, change it to:
DISKNM=/work2/01118/tg803972/frontera/RT instead of DISKNM=/work2/01118/tg803972/frontera/RT/NEMSfv3gfs/

[ayumifuji]

You're right! Now the RT (coastal_ike_shinnecock_atm2sch) ran successfully. Thank you!!

[odotyy]

Hi all,

I am also attempting the rt.sh script but am running into an error in "compile_11_intel" because of a missing file "fv3_11_intel.exe". I believe this error is probably related to the following issues seen in "/scratch1/09221/odoty/frontera/odoty/FV3_RT/rt_241696/compile_11_intel/err"

1. "Platform 'frontera.intel' configuration file does not exist"

2. "CMake Error at CMakeLists.txt:157 (find_package):
   By not providing "FindNetCDF.cmake" in CMAKE_MODULE_PATH this project has asked CMake to find a package configuration file provided by "NetCDF", but CMake did not find one.
   Could not find a package configuration file provided by "NetCDF" (requested version 4.7.4) with any of the following names:
   NetCDFConfig.cmake
   netcdf-config.cmake
   Add the installation prefix of "NetCDF" to CMAKE_PREFIX_PATH or set "NetCDF_DIR" to a directory containing one of the above files. If "NetCDF" provides a separate development package or SDK, be sure it has been installed."

I have checked that the spack-stack netcdf-c and netcdf-fortran modules are loaded after loading the ufs_frontera modules.

Thanks!
Olivia

[uturuncoglu]

@odotyy I am getting permission issue. Please check the log file like me and post it end of the file if you are seeing any error message etc.

[odotyy]

Hi Ufuk,

Here is the whole log file:
compile_11_intel.log

I think you should have permissions now, sorry about that.

Thanks,
Olivia

[uturuncoglu]

@odotyy Can you also share the err file under /scratch1/09221/odoty/frontera/odoty/FV3_RT/rt_75079/compile_11_intel.

[uturuncoglu]

@odotyy you might also want to run module purge command before running RT.

[odotyy]

This is the err file, I will try the module purge and see if anything happens.

err.txt

[pvelissariou1]

@odotyy Are you running the rt.sh script inside conda? If yes, this might be the culprit. Had similar issues in the past when running tasks like that inside a conda environment.

[odotyy]

I haven't activated my conda environment but it is running within my miniconda3 folder. I have also tried the rt.sh script with the ufs_coastal directory outside of the miniconda3 folder, but it shows an issue with undefined reference to "netcdf_mp_nf90".

[pvelissariou1]

@odotyy Ok, let's see what Ufuk finds out. From our side these RTs work on Frontera as expected. Did you load the modules manually?

module use YOUR_UFS_COASTAL_DIR/modulefiles
module load ufs_frontera.intel
module list

to check if you are getting any warnings?

[uturuncoglu]

@odotyy maybe your existing environment (loaded modules) is interfering with the spack-stack ones. So, module purge could fix the issue. I have just compiled the case without any issue.

[uturuncoglu]

@odotyy In my case I have following modules loaded,

1) intel/19.1.1   2) impi/19.0.9   3) git/2.24.1   4) autotools/1.2   5) python3/3.7.0   6) cmake/3.24.2   7) hwloc/1.11.12   8) xalt/2.10.34   9) TACC

[pvelissariou1]

@uturuncoglu @odotyy I believe a check in the logs in tests/log_frontera may give some hinds

[pvelissariou1]

@uturuncoglu I think it will be a good idea to have module purge in all modulefiles in ufs-coastal (I do the same in CoastalApp)

[uturuncoglu]

@pvelissariou1 If it works. I could added to the frontera one.

[pvelissariou1]

@odotyy Please remove TACC module from your .bashrc (important). Or use module purge as Ufuk suggested.

[odotyy]

I have just tried that and then attempted to run it again and still got the same errors as before

[uturuncoglu]

@odotyy I am seeing Platform 'frontera.intel' configuration file does not exist in the error file but before that it seems that the module are loaded. The complains about netcdf. It is strange. Probably related with the environment.

[uturuncoglu]

@odotyy The jobs are using flex and mine is waiting in the queue. I'll update you once it runs.

[odotyy]

Okay, thank you both for your help so far.

[pvelissariou1]

@uturuncoglu Yes in Frontera one. We might need to let the UFS people to include it in all modulefiles

[odotyy]

@pvelissariou1 @uturuncoglu My compilation ran this morning! Not sure what was changed, but thanks for your help!

[uturuncoglu]

@odotyy That is great. I am not sure why but Frontera is acting weird. Anyway, let us know if you need any further help.

[uturuncoglu]

@odotyy Please also fix your permission. So, I could access to your run directory.

[uturuncoglu]

@memarisa It seems that there is an issue in your configuration and it might be related with your job submission script. In schsim side some PETS are finalizing but others not. Maybe it is just issue with resource allocation. If I could access to your job submission script i could try to run in my side and fix the issue.

[memarisa]

Yes that's right. I want to prevent idle time. However, if I increase the # of cores for OCN component to something like 16 and the rest to 11, this causes the model to crash completely. I could only make it work as long as I have the same # of cores for each component.

[uturuncoglu]

@memarisa Okay. Let me work on it and share the job submission script with you. Maybe we could discuss more in next meeting.

[memarisa]

@uturuncoglu That would be great. Thank you!

[uturuncoglu]

@memarisa The case is working in my end. I did couple of modıfıcation in param.nml since there were some missing parameters. I wonder if you are using recent version of the model or not. If not, please update your version to work with the run directory that I modified. I did 3-hours of simulation and checked the ESMF_Profile.summary. The total run with default layout is around 348 sec. I also removed mediator history and restart phases from your run sequence to reduce I/O botlleneck. Based on profile, the main time spend on OCN->MED connector. I increase the number of cores in OCN change the mediator pets to cover all the components.

My run directory is in /work2/01118/tg803972/frontera/memarisa.

But, when I increase the number of pets I am getting following error,

0: ABORT:  STEP: wetted cross section length on open bnd <=0; boundary ndx=           3 , length=  0.000000000000000E+000

I check the mesh and also incoming pressure field with ParaView.

Everything seems fine. Maybe somebody else with more experience with SCHSIM could help at this point. Maybe there are some parameters in SCHSIM side for the decomposition. It seems that in your case the mesh is extending through the river etc. So, that could be source of the issue. @yunfangsun @josephzhang8 Do you have any idea?

[uturuncoglu]

@memarisa @josephzhang8 @yunfangsun The case is running if I assign 12 core to SCHSIM but fail with 48.

[josephzhang8]

I'll be happy to help. The dir:
/work2/01118/tg803972/frontera/memarisa

does not have all SCHISM inputs, e.g. sflux. Can someone explain?

[josephzhang8]

Just realized that the atmos forcing came from the cap, so there is no sflux. So I'd suggest looking at the cap outputs carefully; or someone can convert the outputs into a nc (nws=4 format, with values interpolated onto SCHISM mesh).

[uturuncoglu]

@josephzhang8 Thanks for your help. Yes, this is DATM coupled configuration and it only uses U, V and P as forcing. I checked the import staten and it is fine but maybe something going on strange where that information is transferred to SCHSIM internal data structures. I agree that looking SCHSIM output might reveal some light about the issue. @memarisa could you try to look at it?

[memarisa]

@josephzhang8 @uturuncoglu Thanks you both for your help. Yes, I am working on it and let you know as soon as I combined the files.

[memarisa]

@josephzhang8 @uturuncoglu The combined output for Ufuk's run is located at this directory:
/scratch3/projects/CATUFS/ufs-util-saeed/combined-output-ufuk
(I couldn't run the .exe file in his directory, so I had to copy it there.)

There are no interpolated variables in the NetCDF file because the model crashed too early, which makes sense since Ufuk intentionally used an uneven number of cores to cause the crash. @uturuncoglu , how can we obtain interpolated values with higher temporal resolution? Currently, the model generates outputs every 5 hours.

When using an equal number of cores, as Ufuk initially did, the model seems to work, at least for the 5-day duration that I checked. The combined outputs from this model, which uses the same input, grid, etc., as those in Ufuk's directory, are located at:
/scratch3/projects/CATUFS/ufs-util-saeed/run0/outputs
I ran it for 24 hours, and here are the interpolated forcing variables that SCHISM generates.

Wind - UFS:

Interpolation outside UFS (using MATLAB with the natural neighbor method for interpolation and nearest for extrapolation)

Air pressure - UFS

Interpolation outside UFS

[memarisa]

I noticed some discontinuities in the interpolated (MATLAB) values for air pressure and wind near the open boundary in the southeast. When plotting the forcing data and the domain, I found that the forcing data does NOT cover the entire domain, leaving a few nodes outside the domain. The previous error referred to nodes #3 and #7, which fall in that area. I suspect this is causing the model to crash. I plan to extend the forcing to cover the entire domain and test it again to see if this resolves the issue.

@uturuncoglu @josephzhang8
My observations suggest that for grid nodes outside the forcing domain, UFS extrapolates the values, and it seems to do this better than the "nearest method." However, this does not guarantee that the model won't crash. Even when the model doesn't crash due to this issue, the SCHISM output (salinity and water temperature) contains NaNs (9.9692e+36), which suggests that either the UFS-extrapolated values aren't being passed to SCHISM, or there's another underlying issue.

[josephzhang8]

@memarisa : yes that should be the cause. We had crash before with atmos grid not covering SCHISM domain. Good catch.

[uturuncoglu]

@memarisa Let us know how it goes and if you need help.

[odotyy]

Hey all, I am still working on the NYH case with DATM/SCHISM. I am looking now at the results from short test runs and have a few questions.

First, I have directly used my standalone SCHISM grid, data, etc. with updated ESMF mesh and atmospheric sflux files in ufs-coastal, however when I look over the results there is a huge amount of 0 salinity water that ends up in the NYH that originates in the Hudson River. I have toyed around with changing the amount of discharge from the Hudson and even set the inflow to be zero, but it still shows up. Could this be related to the DATM files/ESMF mesh/sflux, or is there some method you’d suggest to go about testing why this happens?

Second, I am trying to look at the horizontal velocity and other so I can try to troubleshoot the issue above, but I can’t seem to get the SCHISM combine output working for variables with ivs =2 in the schism old I/O output, is there some other method of visualization that is better compatible with ufs-coastal/weather?

The path to the run folder I am using is /work2/09221/odoty/frontera/ufs-weather-model/tests/RUN

Thanks,
Olivia

[josephzhang8]

@odotyy: combining the old I/O requires the stack is complete (i.e. no partial combine).

[odotyy]

Good to know, thank you!

[josephzhang8]

You can always adjust stack size to force the outputs.

[odotyy]

Okay, I will try that as well. I am using the combine_output11_MPI script to try to visualize Horizontal velocity but am struggling with getting it working since it is a 4 dimensional variable. Could you help me with the syntax of how that would work for a variable like this? Using the "combine_output11_MPI -n 44 -b 1 -e 31 -w 1 -v hvel -o hvel" does not seem to make it able to be visualized in schismview

[josephzhang8]

I'd just use:

combine_output11_MPI -n 44 -b 1 -e 31 -w 1

[OrionScharton]

Hey all,

I have ran into some trouble compiling FVCOM (v4.4.2). I needed to change the compiler I use and when I tried to recompile I have ran into the error shown below. Would anyone have any ideas where this is coming from, I have not encountered this error before. The compiler being used is the newer intel compiler (2024).

Thanks,
Orion

[uturuncoglu]

@OrionScharton Can you post also steps that you are following. so, maybe we could reproduce it in our end. BTW, which platform are you using? Why did you need to change compiler and also are you compiling code as a part of UFS Coastal or this is just a standalone model.

[sebinjohn111]

Hi Ufuk,
I would like some help in setting up the ufs-weather model for New York Harbor as I am having difficulties running it.
I managed to run the regression test for the FVCOM in the Scituate Harbor (please see the attachments) and attempted to create a similar setup for the New York Harbor. However, the model encounters issues with NAN values (after 10 minutes of run).
The files are located as follows:

1. SCITUATE_UFS: /scratch3/projects/CATUFS/sebin/Round3/SCITUATE_UFS
2. NYH_UFS: /scratch3/projects/CATUFS/sebin/Round3/NYH_UFS
   UFS_Scituate_sebin (2).pdf

[memarisa]

Hi Ufuk @uturuncoglu

I have a question regarding two-way coupling functionality in UFS. Specifically, I ran tests using the atm2sch2ww3 configuration with the following compile command on Frontera:

./compile.sh frontera "-DAPP=CSTLSW -DPDLIB=ON -DUSE_ATMOS=ON -DNO_PARMETIS=OFF -DOLDIO=ON" "sch_WW3" intel YES NO

Based on my understanding, this configuration should enable two-way exchange of information between the wave model (WW3) and the hydrodynamic model (SCHISM). However, my results suggest that UFS currently incorporates wave data into the hydrodynamic calculations (SCHISM receiving from WW3) but does not allow feedback from the hydrodynamics to the wave model (WW3).

I have attached two figures to show this: the first figure shows the effect of waves on the hydrodynamic variables (e.g., water temperature, salinity, and velocities), while the second figure shows no reciprocal effect of hydrodynamics on the wave variables (e.g., peak frequency and significant wave height).

Could you clarify if UFS currently supports fully two-way coupling, or if this functionality is something that could be considered for future development?

Thank you for your help.

Figure 1

Figure 2

[uturuncoglu]

@memarisa As I know you also need to activate couple of options in the ww3 specific namelist file to allow feedback from ocean. At this point, we are only accepting u and v components of current from ocean. I just want to make sure that you have set those options. For your reference you need to have following options in your ww3_shel.nml,

  input%forcing%winds      = 'C'
  input%forcing%currents   = 'C'

[memarisa]

You were right! The input forcing for currents was off. I made the adjustment and ran a test case, and I can see now the effect of currents on waves. Thank you for you help @uturuncoglu

[uturuncoglu]

@memarisa That is great. Let us know if you need any further help.

[BrendanGramp]

Hi Ufuk,

I am attempting to follow the steps in the wiki for developing new configurations for UFS-DATM SCHISM WW3 (while adapting the information for the UFS-DATM WW3 RT I'm using.)

When checking the 'wind_atm_fin_ch_time_vec_STR_fixed.nc' file, the times given in the file appear to be given every 5 hours from 2008/08/23 00:00:00 to 2008/10/21 19:00:00. This also appears to be the case from the 'ww3_prnc.out' file from the UFS WW3 regression test, a portion of which is pictured below.

I want to confirm if this is the case to ensure that I understand the components of the .nc file correctly so that I can create one matching my own configuration and since seeing 5 hour intervals for the wind forcing data makes me think I'm not understanding it correctly.

[uturuncoglu]

@BrendanGramp I am not sure about the source of that forcing since I did not create it. You could also use cod sinfo command in that file to see its time information. You don't need to follow same structure. Once you have same variable with same units in your file (be sure that covers your domain), you could have higher temporal resolution input. The only think that you need to be careful is to use ESMF Mesh file (used by CDEPS) consistent with your input data fine in terms of the definition of domain. There are different ways to create ESMF Mesh file, so let me know if you need help about it.

[BrendanGramp]

@uturuncoglu I could not find any information regarding cod sinfo so I am guessing that you meant cdo sinfo. I ran that on the .nc file and it also appears to be indicating 5 hour intervals for the input data. I've attached a snippet of the output from running cdo sinfo below.

[uturuncoglu]

yes you are right. sorry for typing.

[BrendanGramp]

Hi Ufuk @uturuncoglu

I have another question regarding the files from the UFS-DATM + WW3 regression test. I am trying to understand how the regression test uses the mesh corresponding to the mesh.shinnecock.cdf5.nc file. The output from the regression test seems to match the node and element count given in the inlet.msh file which is used in the ww3 files. The .cdf5.nc file is used in the ufs.configure file, but I'm not sure how that is read in by the regression test since the output corresponds to the node and element count in the inlet.msh file.

[uturuncoglu]

@BrendanGramp The mesh is used in WW3 coupling interface mainly for creating mesh in the coupling layer. But, we found out that the SCRIP grid definition file (used to create ESMF mesh file) created by ww3_grid too has some issue and does not producing each mesh with the text file given to WW3 (like inlet.msh). You could find more information in the following link, NOAA-EMC/WW3#1319