Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KPP standalone interface from Obin Sturm #2482

Merged
merged 24 commits into from
Nov 21, 2024
Merged

KPP standalone interface from Obin Sturm #2482

merged 24 commits into from
Nov 21, 2024

Conversation

lizziel
Copy link
Contributor

@lizziel lizziel commented Sep 26, 2024

Name and Institution (Required)

Name: Lizzie Lundgren
Institution: Harvard University

Describe the update

This draft PR contains updates for KPP standalone interface from Obin Sturm (GMAO) contained in GEOS-ESM#9. That original PR is built off of branch geos/develop in the GEOS-Chem fork in GMAO's GitHub. That version is not mergable into the standard model so this PR will be used instead.

Tagging @obin1, @yantosca

Expected changes

I have pushed several updates to this PR:

  1. kpp_standalone_interface.yml configuration file:

    • Comments have been updated
    • This is now always copied to any fullchem run directory (GCClassic only).
    • The following YAML tags have been added. You now must manually activate the KPP standalone interface. Also, you can set the interval during which model state will be archived. By default this is set to the interval 1900..2100, but you can manually restrict this to e.g. the last timestep of a run.
    settings:
      activate: true                       # Main on/off switch
      start_output_at: [19000101, 000000]  # Save model state for KPP standalone
      stop_output_at:  [21000101, 000000]  # ... if between these 2 datetimes
  2. GeosCore/kpp_standalone_interface.yml has been renamed to GeosCore/kppsa_interface_mod.F90. All derived types, variables, and public-facing routines have been renamed to use the prefix KppSa_, for consistency. Derived-type
    object Kpp_Standalone_YAML has new been renamed to KppSa_State for brevity.

  3. We have added a new derived-type object KppSa_ActiveCell to hold the ActiveCell logical flag and Active_Cell_Name fields. This is declared with !$OMP THREADPRIVATE because it is updated within the main parallel loop within the DO_FULLCHEM routine.

  4. In the KppSa_Write_Samples routine (formerly Write_Samples), we have placed the I/O code within an !$OMP CRITICAL block. This will prevent other threads from trying to write to the file, which can cause incomplete and/or corrupted files.

  5. We have added a new function KppSa_Check_Time to determine if we are within the time window for archiving model state specified in the YAML file. This updates a new flag KppSa_State%SkipWriteAtThisTime, which is used to determine if we need to skip writing output on this timestep.

  6. We now print the following output at initialization:

===============================================================================
KPP STANDALONE INTERFACE

Model state will be archived at these sites:

LosAngeles               ( -118.2430,   34.0522 )
McMurdo                  (  166.6698,  -77.8455 )
Paris                    (    2.3522,   48.8566 )
Beijing                  (  116.4074,   39.9042 )
Kinshasa                 (   15.3105,   -4.3033 )
Kennaook                 (  144.6833,  -40.6833 )
Graciosa                 (  -28.0069,   39.0525 )
Utqiagvik                ( -156.7886,   71.2906 )
Ozarks                   (  -91.2590,   37.5020 )
Amazon                   (  -62.2159,   -3.4653 )
Congo                    (   12.5484,   -5.9175 )
Borneo                   (  114.0000,    0.0000 )
IndianOcean              (   87.2000,   23.0000 )
AtlanticOcean            (  -41.5748,   34.7079 )
PacificOcean             ( -121.9645,    0.0000 )
ElDjouf                  (   -6.6661,   21.5008 )

For GEOS-Chem vertical levels:
   1   2  10  23  35  48  56
Starting at 20190701 004000
Ending at   20190701 010000
===============================================================================

Reference(s)

None

Related Github Issue

@lizziel lizziel added category: Feature Request New feature or request no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations Topic: Fullchem Simulations Related to full-chemistry simulations labels Sep 26, 2024
@lizziel lizziel requested a review from yantosca September 26, 2024 19:29
Signed-off-by: Lizzie Lundgren <[email protected]>
@obin1
Copy link

obin1 commented Sep 26, 2024

Hi Lizzie, thanks for getting this started! I'm still working on redoing a few c360 runs with the new compute resources on Discover before we can write up the manuscript about this tool and species-level tolerances. I will keep you all posted when that's out as a preprint, and thanks again for moving this PR upstream

@yantosca yantosca added this to the 14.5.1 milestone Sep 26, 2024
@yantosca yantosca self-assigned this Sep 27, 2024
@yantosca yantosca added the topic: Chemical Mechanisms Related to KPP and/or GEOS-Chem chemistry mechanisms label Sep 27, 2024
@yantosca yantosca modified the milestones: 14.5.1, 14.5.0 Sep 27, 2024
Copy link
Contributor

@yantosca yantosca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@obin1: Also not related to this PR, but if you have any python scripts to read & plot output from the box model we shoudl think about bringing those into GCPy.

GeosCore/CMakeLists.txt Outdated Show resolved Hide resolved
run/shared/kpp_standalone_interface.yml Show resolved Hide resolved
@lizziel lizziel changed the base branch from dev/14.5.0 to dev/no-diff-to-benchmark September 27, 2024 18:18
run/shared/kpp_standalone_interface.yml
- Fixed typo: "output_dir" -> "output_directory"

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/fullchem_mod.f90
- Use keyword arguments, for clarity

GeosCore/kpp_standalone_interface.F90
- Remove Active_Cell and Active_Cell_Name from KPP_Standalone_YAML
- Add new derived type KPP_Standalone_ActiveCell_Type, which is now
  declared with !$OMP THREADPRIVATE.  This contains the Active_Cell
  and Active_Cell_Name fields.
- Added new variable KPP_Standalone_ActiveCell, based on
  new KPP_Standalone_ActiveCell_Type

NOTE: The KPP_Standalone_YAML is initialized outside of a parallel
loop, so it does not need to be declared !$OMP THREADPRIVATE.  But the
choice of whether an (I,J,L) location corresponds to one of the
"active_cells" (listed in the kpp_standalone_interface.yml file) happens
within a parallelized loop.  Thus we need to move the Active_Cell and
Active_Cell_Name fields out of KPP_Standalone_YAML and into
KPP_Standalone_ActiveCell.

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca
Copy link
Contributor

@obin1 @lizziel: See the updates in my bugfix branch below. This fixes the parallelization issue listed above.

run/GCClassic/createRunDir.sh
- Copy run/shared/kpp_standalone_interface.yml to fullchem rundirs

run/shared/cleanRunDir.sh
- Skip removing bpch files (and {diag,tracer}info.dat files), we no
  longer generate bpch output
- Add comments and usage examples
- Remove all fort.* files
- Remove OututDir/*.txt files as well (these are KPP standalone
  interface files)

CHANGELOG.md
- Updated accordingly

Signed-off-by: Bob Yantosca <[email protected]>
.gitignore
- Also ignore *.exe files

GeosCore/fullchem_mod.F90
- Remove State_Grid from call to Check_ActiveCell

GeosCore/kpp_standalone_interface.F90
- Updated subroutine header comments
- Added cosmetic changes for clarity (mostly making code fit within
  80 characters, for better readability)
- Removed Active_Cell and Active_Cell_name from the
  KPP_Standalone_Interface_Type (should have been done previously)
- Remove State_Grid argument from Check_ActiveCell, this was only
  used for debugging.   Also removed commented out debug prints.
- Use Format statements with write statments where expedient
- Echo a message when the KPP standalone interface is manually disabled
  (i.e. when settings:activate = false)
- Added display of location names and lon/lats at end of the routine
  Config_KPP_Standalone
- In routine Write_Samples, wrap file I/O in an !$OMP CRITICAL block,
  in order to prevent more than one thread from writing to each file.
- Removed RETURN statement from within !$OMP CRITICAL block, this
  is not allowed.
- Bug fix: Write Kpp_Standalone_ActiveCell%Active_Cell_Name to file,
  and not Kpp_Standalone_YAML%Active_Cell_Name (which has been removed).

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kpp_standalone_interface.F90
- Moved to kppsa_interface_mod.F90

GeosCore/kppsa_interface_mod.F90
- Moved from kpp_standalone_interface
- Renamed types, variables and routines using "KppSa" prefix:
  - KPP_Standalone_Interface_Type -> KppSa_Interface_Type
  - KPP_Standalone_ActiveCell_Type -> KppSa_ActiveCell_Type
  - KPP_Standalone_YAML -> KppSa_State
  - KPP_Standalone_ActiveCell -> KppSa_ActiveCell
  - Check_Domain -> KppSa_Check_Domain
  - Check_ActiveCell -> KppSa_Check_ActiveCell
  - Config_KPP_Standalone -> KppSa_Config
  - Write_Samples -> KppSa_Write_Samples
  - Cleanup_KPP_Standalone -> KppSa_Cleanup
- Added KppSa_Check_Time function to determine if it is time to
  write KPP standalone output.  This allows you to only write output
  e.g. at the end of a run instead of for each timestep.
- Added KppSa_State%SkipWriteAtThisTime field, which is used to
  determine if we need to exit a routine early.

GeosCore/fullchem_mod.F90
- "USE Kpp_Standalone_Interface" -> "USE KppSa_Interface_Mod"
- Call renamed routines from kppsa_interface_mod.F90
- Now call KppSa_Check_Domain only if it is the first call to
  DO_FULLCHEM.  This is to avoid repeated computations.
- Now call KppSa_Check_Time to determine if we are in the time window
  when the model state should be archived to disk.
- Updated comments and comment headers

run/shared/kpp_standalone_interface.yml
- Added "start_output_at" to denote starting date time for archiving
  model state
- Added "stop_output_at" to denote ending date time for archiving
  model state
- Updated comments

CHANGELOG.md
GeosCore/CMakeLists.txt
- Updated accordingly

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kppsa_interface_mod.F90
- In routine KppSa_Check_Time:
  - rRmoved leftover debug print statements
- In routine Kpp_Check_ActiveCell
  - Added an IF statement to exit after setting KppSa_State%Active_Cell to .FALSE.
    and  KppSa_Active_Cell_Name to '' if we are outside of the time
    window specified in the kpp_standalone_interface.yml fiel.
  - This will ensure that we only archive model state to disk during
    the specified time window, which helps with computatonal efficiency.
- In routine KppSa_Config:
  - Now write starting & ending date of archival window to log file

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca marked this pull request as ready for review October 8, 2024 19:51
run/GCHP/createRunDir.sh
- Added an if statement to copy run/shared/kpp_standalone_interface.yml
  to GCHP fullchem run directories (any option)

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca
Copy link
Contributor

yantosca commented Oct 9, 2024

@lizziel, @obin1: I have confirmed that this PR works in GCHP. I was able to restrict output to the last timestep of a 1-hr GCHP run and got a bunch of files timestamped at 00:40z.

@yantosca yantosca requested a review from obin1 October 9, 2024 19:09
@yantosca yantosca added this to the 14.6.0 milestone Oct 10, 2024
@yantosca
Copy link
Contributor

@obin1 @lizziel: FYI I have also modified this PR to save out the ICNTRL and RCNTRL solver options to the output file. This is necessary in case we want to use different solvers.

Now that we have per-species ATOL settings in GEOS-Chem 14.5.0, I will add another fix to save that out. This will be needed to set the ATOL for family/dummy species to a high value so as to keep them out of the error norm computations.

GeosCore/fullchem_mod.F90
- Pass KPP integrator arguments ICNTRL and RCNTRL to KppSa_Write_Samples
  so that we can include it in the KPP standalone output.

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/CMakeLists.txt
- Changed permission from chmod 755 to chmod 644

GeosCore/kppsa_interface_mod.F90
- Now accept ICNTRL and RCNTRL as inputs to routine KppSa_Write_Samples
- Change number of header lines from 48 to 60
- Better separate Meteorological metadata and KPP parameters
  for readability
- Write out ICNTRL as 2 lines of format 10i6
- Write out RCNTRL as 4 lines of format 5f13.6
- Changed format e25.16e3 to es25.16e3, which changes the output from
  e.g. 0.5e1 to 5.0e0, etc., which is more standard
- Write out ATOL after the species concentration with format es10.2e2
- Cosmetic changes (indentation, comments)

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kppsa_interface_mod.F90
- In routine KppSa_Write_Samples:
  - Added a "," in between the concentration and abs tolerance output
  - Simplified FORMAT statements 120 and 130
  - Added FORMAT statements 140 and 150

Signed-off-by: Bob Yantosca <[email protected]>
GeosCore/kpp_standalone_interface.F90 Outdated Show resolved Hide resolved
GeosCore/kpp_standalone_interface.F90 Outdated Show resolved Hide resolved
@yantosca
Copy link
Contributor

All GCClassic integration tests passed:

==============================================================================
GEOS-Chem Classic: Execution Test Results

CodeDir   : 0ecd8fa GEOS-Chem update: Merge PR #2507 (Fix CEDS TMB emissions)
GEOS-Chem : a86303771 Merge PR #2482 (KPP standalone interface from Obin Sturm)
HEMCO     : deaa192 HEMCO 3.10.0 release
Cloud-J   : f8a2b7f Update version number for 8.0.1 release
HETP      : 2a99b24 Merge pull request #2 from geoschem/bugfix/initialize_local_variables

Using 24 OpenMP threads
Number of execution tests: 30

Submitted as SLURM job: 58916604
==============================================================================

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Also all tests were zero-diff w/r/t 14.6.0-alpha.0 except for

  • TOMAS15 (parallelization error?)
  • APM (parallelization error?)

@yantosca
Copy link
Contributor

All GCHP integration tests passed:

==============================================================================
GCHP: Execution Test Results

CodeDir       : 15de6b3 GEOS-Chem update: Merge PR #2507 (Fix CEDS TMB emissions)
MAPL          : 9ad63ae Merge PR #37 containing update to vertically flip imports with dimensionless pressure proxy lev coordinates
GMAO_Shared   : 4ddb3ec Merge pull request #2 from geoschem/feature/mapl-upgrade
ESMA_cmake    : ad5deba Added ecbuild as a submodule of ESMA_cmake
gFTL-shared   : 4b82492 Merge branch 'upstream_v1.5.0' into feature/v1.5.0
FMS           : 259759d Merge pull request #3 from geoschem/feature/update_gmao_libs
FVdycoreCubed : af42462 Merge PR #8 (Add PLEadv diagnostic for offline advection in GCHP)
geos-chem     : a86303771 Merge PR #2482 (KPP standalone interface from Obin Sturm)
HEMCO         : deaa192 HEMCO 3.10.0 release
yaFyaml       : 19afe50 Merge branch 'upstream_v1.0.4' into feature/v1.0.4
pFlogger      : 2c4b724 Merge branch 'upstream_v1.9.1' into feature/v1.9.1
Cloud-J       : f8a2b7f Update version number for 8.0.1 release
HETP          : 2a99b24 Merge pull request #2 from geoschem/bugfix/initialize_local_variables

Number of execution tests: 12

Submitted as SLURM job: 58926494
==============================================================================

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

All tests were zero-diff w/r/t 14.6.0-alpha.1

@yantosca yantosca merged commit a863037 into dev/14.6.0 Nov 21, 2024
@yantosca yantosca deleted the feature/kpp_box branch November 21, 2024 15:33
@yantosca yantosca restored the feature/kpp_box branch November 21, 2024 21:28
yantosca added a commit that referenced this pull request Nov 22, 2024
This merge brings PR #2482 (KPP standalone interface from Obin
Sturm, by @lizziel) into the GEOS-Chem 14.5.1 development stream.

This PR brings the interface for the KPP standalone box model
into GEOS-Chem 14.6.0.  A new kpp_standalone_interface.yml file
is placed into the run directory.  In this YAML file, the user
can select the locations and times for archiving model state
to text files that can be fed into the KPP standalone model.

Also the writing of files is now done in a thread-safe manner.

Signed-off-by: Bob Yantosca <[email protected]>
@yantosca yantosca deleted the feature/kpp_box branch December 2, 2024 20:45
@lizziel lizziel modified the milestones: 14.6.0, 14.5.1 Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Feature Request New feature or request no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations topic: Chemical Mechanisms Related to KPP and/or GEOS-Chem chemistry mechanisms Topic: Fullchem Simulations Related to full-chemistry simulations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants