Replies: 6 comments 14 replies
-
|
Can git handle tracking different versions of boundary conditions? |
Beta Was this translation helpful? Give feedback.
-
I guess so! If we start having those But if the metadate has the source code info (as I suggested above), then may be we don't need to have the files versioned. |
Beta Was this translation helpful? Give feedback.
-
|
A particular set (version) of the land boundary condition files should, for the most part, be compatible with both the AGCM and the (coupled) AOGCM. (There have been some changes in the required files and file formats across major GCM versions in the past, but this shouldn't be an issue in practice right now.) The source that generates bcs is already tagged with the GCM. However, there are configuration choices in this bcs-producing code that also determine the science content of the bcs. That is, to uniquely identify a specific set of land bcs, it's not enough to just know the tag of the source code that generated them. Moreover, each bcs version comes in a range of resolutions, with the higher-resolution files being quite large, so I don't think we could put the bcs files themselves into git. The only workable solution that I can see is to maintain the bcs in a central directory, with each version stored in its own subdirectory and (ideally) with some README documentation. As far as I know, this dir is still /[dnb]/ltakacs/bcs/, but I'm not sure Larry is still responsible for maintaining the bcs archive. FYI, this bcs discussion reminds me of a related issue re. the organization of the "mk_restart" source code within the GCM: |
Beta Was this translation helpful? Give feedback.
-
@gmao-rreichle I was under the impression (from @biljanaorescanin via modeling meeting) that there was going to be a re-org of that code, perhaps into this repo or https://github.com/GEOS-ESM/GMAO_Shared I am glad you brought it up. 👍 Following along the suggestion
|
Beta Was this translation helpful? Give feedback.
-
|
@gmao-rreichle would you please expand on
|
Beta Was this translation helpful? Give feedback.
-
|
I don't know if there's already a clean (set of) rc file(s) that could be committed. I don't have experience running the bcs generation code. Sarith would know. If this doesn't already exist, it probably isn't too difficult to add. |
Beta Was this translation helpful? Give feedback.
-
|
Ah! True! There are those input ancillary data sources. We would have to 🤔 how to organize them as well. Some of those are external to our org! |
Beta Was this translation helpful? Give feedback.
-
|
Some details follow reg how BCs are generated from @smahanam
The script is not yet fully ready for tripolar grids. Two things need to be done:
In the meantime, from your build, you can just try this command to get an idea how this script works. Reg. directory structure, use following: From @yvikhlya:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Tom, Since I am operating at a lower level, I thought it is not appropriate for me to share my thoughts here. Now that I heard you are making some plans to create a data repo, I thought you might find below information is useful ...................................................................... As I understood. AOGCM can use MAPL_Tripolar.nc to specify the MOM version. I suggest AOGCM could (rather should) still use all non-Ocean input files from the centralized BCs directory at /discover/nobackup/ltakacs/bcs (for NCCS, there is a copy somewhere on NAS too) – either by copying or linking (preferably). As you know, /discover/nobackup/ltakacs/bcs directory is tag-based, version controlled and reproducible. In this way, outside the ocean, AOGCM ocean will always be linked to a modeling group controlled AGCM BCs data set (tiles and AGCM input files).
create ‘til’ and ‘rst’ directories and create links to respective files from Larry’s directory.
This would create the master til file that AOGCM needs. Then, link everything else from Larry’s centralized directory - topo_DYN, topo_GWD, topo_GRB, visdf, nirdf, lai, green, ndvi, vegdyn, clsm etc Briefly, Larry has structured the /discover/nobackup/ltakacs/bcs/ directory based on the tag that he used to create BCs files. Then enter ‘NL’ series: We laid a brand new land surface in around 2015 or so using new global data sets on finer scale tiles. There is a Tech Memo that is a good document to refer to if one needs details. NL uses a digital version controlling system, NL, NLv2, NLv3 and NLv4 – we continue to use ‘Icarus’ as the family name to remember the legacy of the incorrigible young fellow that died many centuries ago. All BCs are reproducible. So for the NL series, if you look in clsm/README in any BCs directory, you will find the tag that would reproduce the BCs directory I am encouraging you to consider using these Modeling group sanctioned, tag-controlled AGCM input files from the centralized directory, rather than attempting to create all those data files by yourselves. Not only, you save time, even though, you don’t know what you are using, somebody else in the modeling group should be able to explain them for you. This is merely what I would do. As I said Shantha, I am operating at a very low level. Planning/designing is beyond my task there are others at higher levels in the hierarchy for that. setenv OMP_NUM_THREADS 1 |
Beta Was this translation helpful? Give feedback.
-
|
Hi Sarith,
My plan is only regarding organization and distribution of input files. I very much want to ensure that any new mechanism for this exactly captures the existing input files. At most I would be copying the existing files, but ultimately I would expect that appropriate individuals will maintain specific subtrees in the new repository. I also envision that there will be a review process as new versions of the files are submitted.
One interesting issue that your discussion does raise though, is that of the “secondary” input files that are derived from the “primary” input files. Is there a reason why these are not all in the centralized directory?
Another question – this is the 2nd time that I’ve heard the centralized directory referred to as “tag-based” and “version controlled”. I get the “tag-based” aspect, but don’t really understand how that directory is version controlled beyond the “tag-based” aspect. Is there some underlying versioning technology involved that I did not see in my quick perusal of the structure?
Thanks,
* Tom
*
From: Sarith Mahanama <[email protected]>
Reply-To: GEOS-ESM/GEOSgcm_App <[email protected]>
Date: Friday, January 29, 2021 at 7:35 AM
To: GEOS-ESM/GEOSgcm_App <[email protected]>
Cc: "Clune, Thomas L. (GSFC-6101)" <[email protected]>, Mention <[email protected]>
Subject: [EXTERNAL] Re: [GEOS-ESM/GEOSgcm_App] Common code and scripts for land and ocean surface boundaries/fields (#193)
Hi Tom,
Since I am operating at a lower level, I thought it is not appropriate for me to share my thoughts here.
Instead, I replied directly to Shantha to describe how his proposal could conflict with the current practice.
Now that I heard you are making some plans to create a data repo, I thought you might find below information is useful
to make your plan well informed.
Thanks ! I understand your plan. OK ! I am not going to worry about adding tripolar grids to make_bcs and to the centralized BCs directory anymore.
As I understood. AOGCM can use MAPL_Tripolar.nc to specify the MOM version. I suggest AOGCM could (rather should) still use all non-Ocean input files from the centralized BCs directory at /discover/nobackup/ltakacs/bcs (for NCCS, there is a copy somewhere on NAS too) – either by copying or linking (preferably). As you know, /discover/nobackup/ltakacs/bcs directory is tag-based, version controlled and reproducible. In this way, outside the ocean, AOGCM ocean will always be linked to a modeling group controlled AGCM BCs data set (tiles and AGCM input files).
This is what I would do. I would write a script (I am sure there must be better ways than this) – if you want, I sure can give you a sample script. (Atanas, please feel free to correct me).
1. Select the BCs version – the modeling group knows what is best for your application, Icarus, Icarus-NL, Icarus-NLv3 or whatever) –
2. Suppose we are trying to create CF0090x6C_TM0120xTM0065 BCs. and we want to use Icarus atmosphere and land
3. Find any CF90 subdirectory in Icarus, for e.g.
/discover/nobackup/ltakacs/bcs/Icarus/Icarus_MERRA-2/CF0090x6C_DE1440xPE0720/
create ‘til’ and ‘rst’ directories and create links to respective files from Larry’s directory.
/discover/nobackup/ltakacs/bcs/Icarus/Icarus_MERRA-2/CF0090x6C_DE1440xPE0720/rst/Pfafstetter.rst
/discover/nobackup/ltakacs/bcs/Icarus/Icarus_MERRA-2/CF0090x6C_DE1440xPE0720/rst/CF0090x6C.rst
/discover/nobackup/ltakacs/bcs/Icarus/Icarus_MERRA-2/CF0090x6C_DE1440xPE0720/til/Pfafstetter.til
/discover/nobackup/ltakacs/bcs/Icarus/Icarus_MERRA-2/CF0090x6C_DE1440xPE0720/til/CF0090x6C.til
3. use your MOM file and run mkMOMAquaRaster
bin/mkMOMAquaRaster.x -x 43200 -y 21600 data/MOM/120x65/grid_spec.nc > /dev/null
4. run CombineRasters to overlay your Aqua on Icarus land
bin/CombineRasters.x -f 0 -t 232000000 TM0120xTM0065 Pfafstetter >/dev/null
5. run CombineRasters to overlay Icarus cubed-atmosphere on the above
bin/CombineRasters.x -t 232000000 CF0090x6C TM0120xTM0065-Pfafstetter
This would create the master til file that AOGCM needs.
Then, link everything else from Larry’s centralized directory - topo_DYN, topo_GWD, topo_GRB, visdf, nirdf, lai, green, ndvi, vegdyn, clsm etc
Briefly, Larry has structured the /discover/nobackup/ltakacs/bcs/ directory based on the tag that he used to create BCs files.
This practice continued through Ganymed-4_0 (MERRA-2 uses this)
Then, Ganymed-4_0 tag also produced Icarus BCs.
Also, until Icarus, bug fixes, compiler differences aside, we used the same global data sets to create all land BCs files. It is just the same mask, same tiles. MERRA2 documentation is a good place to look for details.
Then enter ‘NL’ series: We laid a brand new land surface in around 2015 or so using new global data sets on finer scale tiles. There is a Tech Memo that is a good document to refer to if one needs details.
https://ntrs.nasa.gov/api/citations/20160002967/downloads/20160002967.pdf
NL uses a digital version controlling system, NL, NLv2, NLv3 and NLv4 – we continue to use ‘Icarus’ as the family name to remember the legacy of the incorrigible young fellow that died many centuries ago.
All BCs are reproducible. So for the NL series, if you look in clsm/README in any BCs directory, you will find the tag that would reproduce the BCs directory
for e.g. grep TAG /discover/nobackup/ltakacs/bcs/Icarus-NLv3/Icarus-NLv3_EASE/SMAP_EASEv2_M36/clsm/README
CVS TAG : Jason-3_0_LANDBCS
I am encouraging you to consider using these Modeling group sanctioned, tag-controlled AGCM input files from the centralized directory, rather than attempting to create all those data files by yourselves. Not only, you save time, even though, you don’t know what you are using, somebody else in the modeling group should be able to explain them for you.
As you see, NLv3 is the last version that used a Jason tag, and ran on SLES11 nodes. Compilers change overtime, that is why Larry et al.. are following tag-based system to produce BCs that are reproducible.
This is merely what I would do. As I said Shantha, I am operating at a very low level. Planning/designing is beyond my task there are others at higher levels in the hierarchy for that.
If you really want to create BCs files yourself, after above step 5) – you will need to execute below 6 lines. mkCatchParam will use the default configuration (which may be similar to the configuration of the last BCs data set it created, but I don’t necessarily guarantee that).
setenv OMP_NUM_THREADS 1
bin/mkCatchParam.x -x 43200 -y 21600 -g CF0090x6C_DE0120xPE0065-Pfafstetter
setenv OMP_NUM_THREADS 20
bin/mkCatchParam.x -x 43200 -y 21600 -g CF0090x6C_DE0120xPE0065-Pfafstetter
chmod 755 bin/create_README.csh
bin/create_README.csh
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FGEOS-ESM%2FGEOSgcm_App%2Fdiscussions%2F193%23discussioncomment-320552&data=04%7C01%7Cthomas.l.clune%40nasa.gov%7Caa6544f70fd540065b5408d8c452649f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637475205445711545%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tplBXTVT%2B%2BV%2B3c7a0hKNYyKotlhFfCGiW8fwv%2FTNlb0%3D&reserved=0>, or unsubscribe<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABPP7YA5ACKYVOZIYVLUDIDS4KTRRANCNFSM4WEI4WNQ&data=04%7C01%7Cthomas.l.clune%40nasa.gov%7Caa6544f70fd540065b5408d8c452649f%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637475205445721501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vfAYELObufJHF0G2czpO%2BjpKt1N9rQKbhJ2KgCoIX2o%3D&reserved=0>.
|
Beta Was this translation helpful? Give feedback.
-
|
Eliminating binary in favor of netCDF is also on my todo list. This is separate from the data repository concern, and would not be retro active. @atrayano is exploring the elimination of tile files using ESMF capabilities. If that is not sufficient, then we'll develop a NetCDF spec for any remaining binary files. (But this would only be for future GEOS releases.) I'm glad you mentioned provenance. I'd not really been thinking of that aspect, but this is a very important aspect. At the very least we could press for a small text file that accompanies each input file that details the provenance. But ideally it would get encoded in the metadata of the NetCDF files. (I know there are some provenance conventions, but I'm not at all familiar with them, and need to learn.) |
Beta Was this translation helpful? Give feedback.
-
|
nc4: are little endian binaries. They follow MAPL_ReadForcing format (like SST files), 14 word header for time interpolation followed by the data record They are on tile space, land would like to keep them in tile-space so ExtData is not an option. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Tom, I posted the workflow in the other thread, As I understood Now your question : " why these are not all in the centralized directory"
Version Control is the first step: Groups present development work to the modeling group, then they decide whether proposed updates are acceptable and decide that the next version could use this new data set in our modeling system. Icarus-NLv3 was approved after some changes to NLv2. We recorded Jason-3_0_LANDBCS as the tag that can reproduce NLv3 The key change between NLv3 and NLv4 is the 2nd field 'Z2CH' vegdyn_IMxJM.dat - Hope I answered.. |
Beta Was this translation helpful? Give feedback.
-
|
OK - so if the "central" directory had true version control and you could play to your heart's content in your own branch, then the secondary files could always reside in the central directory, right? Let me know if I've not understood correctly. I think you are also confirming that this is not Version Controlled. It is only tag-based. |
Beta Was this translation helpful? Give feedback.
-
|
Yes ! you are right ! |
Beta Was this translation helpful? Give feedback.
-
I would be so happy to do so, but in the past this did not always work. I ended up having a copy of green, ndvi, lai, vegdyn, clsm, rst in coupled boundary conditions directory for each grid which we use because some utils (restart regridding or something) required this. If we could really use all non-Ocean input files from the centralized BCs directory, this would remove so much pain from our heads. |
Beta Was this translation helpful? Give feedback.
-
|
I apologize Yuri ! That overwriting version controlled vegdyn_IMxJM.dat was my idea, I insisted and Larry agreed - I wasn't aware of your copy. Larry had just created Icarus. I had a load of updates to the source codes Surface, land, catch, catchcn and veg - the veg change needed to change vegdyb_, it was a month before some organization changes in the land group, so I just wanted to complete everything and hand over GEOSsurface_GridComp. |
Beta Was this translation helpful? Give feedback.
-
|
Sarith, no need for apologies, it is all work in progress here. But anyway, if we would be confident that coupled model can use land and other BC files from centralized BC directory, that would be great. Absolutely support this. |
Beta Was this translation helpful? Give feedback.
-
|
yes, of course! ocean/sea ice would go with @GEOS-ESM/ocean-team and likewise for others... that way we would have a good chance of minimizing, if not eliminating problems! |
Beta Was this translation helpful? Give feedback.
-
Introduction
The GCM, regardless of whether it runs coupled or uncoupled requires boundary conditions from the
visdf.dat, nirdf.dat, vegdyn.data, lai.data, green.data, ndvi.data)Problem
I am almost 💯 % sure that land files do not differ between coupled and uncoupled versions, which follows from this discussion
More details at GEOS-ESM/MAPL#663 (comment)
However, it is clear that the land files in the coupled model directory are different from the uncoupled files, also from:
Talking to @yvikhlya apparently this has happened in the past, i.e., certain updates happen to the land files and the coupled files do not get sync'ed.
Proposed fix
ocean, sea ice, land, land ice, ocean biology(kPAR - which is currently under ocean now as with @yvikhlya). Perhaps the SI Team (@tclune , @mathomp4 , ... ) will know better.binaryfiles withncfiles which would havemetadata(version number from above 2 and date generated).GEOSgcm_App/gcm_run.j
Lines 351 to 423 in ee1397d
Please contribute to this idea so we can make our code simpler and easier to maintain and run, Thank you!
@yvikhlya @zhaobin74 @amolod @wmputman @sdrabenh @lltakacs
@smahanam @biljanaorescanin @rdkoster @gmao-rreichle
@mathomp4 @atrayano @bena-nasa @tclune
@adarmenov
Beta Was this translation helpful? Give feedback.
All reactions