-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add MOM6 support (om4 025jra ryf) #258
Merged
Merged
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
1d040c7
Factor out the 'static' frequency tag as a variable
marc-white cb8b017
Initial creation of AccessOm4Builder and test data structure
marc-white 7e14a52
Correct exclude pattern, add Om4Builder to test data suite
marc-white 2428c2d
Improved OM4 test data (still doesn't work though)
marc-white a13de8b
Fixed test data, fixed filename regexp
marc-white ed2ba42
Add FIXME for timestamp part of PATTERN regex
marc-white 463005f
Add test_builder_parser tests for OM4
marc-white 6829eb0
Add test_parse_access_filename tests for OM4
marc-white 6796f42
Final tests for AccessOM4 builder (plus another TODO)
marc-white 138e2a6
Add new pattern for OM4; add ability to have multiple redacted 'times…
marc-white 79763cc
OM4 test expansion
marc-white 3212a99
Added test data for panan-01-zstar data
marc-white b6e9d41
Refactor AccessOm4Builder --> Mom6Builder
marc-white 2956b7b
Merge remote-tracking branch 'origin/main' into 175-data-request-add-…
marc-white 0d476c6
Add first MOM6 datasets
marc-white 24d97eb
Merge remote-tracking branch 'origin/main' into 175-data-request-add-…
marc-white b3ab3bd
Merge branch 'main' into 175-data-request-add-om4_025jra_ryf
marc-white 120d92c
Merge branch 'main' into 175-data-request-add-om4_025jra_ryf
marc-white 720cad1
Add test print for debugging
marc-white 3fa73e7
Remove asset print debug
marc-white fc582ce
Add better error statement to CatalogManager __init__
marc-white 6e7a0e9
Expand out ParserError message in validate_parser
marc-white fa89393
Tweak ParserError message
marc-white d428726
Further expand ParserError message
marc-white bf9450d
Add first pass at MOM6 translator (hack job)
marc-white 6f31a53
Next pass at MOM6 translator
marc-white a95194c
Remembered to change the configured Translator
marc-white 0eb2a97
Try MOM6 translator again...
marc-white 581e9ff
Merge remote-tracking branch 'origin/main' into 175-data-request-add-…
marc-white b7856aa
Fix up MOM6 builder after last merge
marc-white 2033537
Update test_builders for MOM6 - new parse_access_ncfile return
marc-white 771978d
Try again without a hard-coded MOM6 model name
marc-white 9e54ff3
Return model translator for MOM6
marc-white b6158ad
Merge branch 'main' into 175-data-request-add-om4_025jra_ryf
marc-white 3531397
Fix MOM6 test variables etc.
marc-white 4aac81b
Added docstring to parse_access_ncfile to make issue clear & updated …
charles-turner-1 6dfbd89
Merge branch 'main' into 175-data-request-add-om4_025jra_ryf
marc-white 47166db
Remove redundant MOM6 translator
marc-white 4331abc
PR updates to builders.py
marc-white 5705d33
Update to except statement in CatalogManager
marc-white 97f18db
Merge remote-tracking branch 'origin/main' into 175-data-request-add-…
marc-white bdb309b
Merge branch 'main' into 175-data-request-add-om4_025jra_ryf
rbeucher 101c948
Ruff fix
marc-white cc2f6e5
Improve test coverage of Builder parser exceptions
marc-white 2637426
Improve manager test coverage
marc-white 428f0d2
Improve builders test coverage
marc-white File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
builder: Mom6Builder | ||
|
||
translator: DefaultTranslator | ||
|
||
sources: | ||
|
||
- metadata_yaml: /g/data/ik11/outputs/mom6-om4-025/OM4_025.JRA_RYF/metadata.yaml | ||
path: | ||
- /g/data/ik11/outputs/mom6-om4-025/OM4_025.JRA_RYF | ||
|
||
- metadata_yaml: /g/data/ik11/outputs/mom6-panan/panant-01-hycom1-v13/metadata.yaml | ||
path: | ||
- /g/data/ik11/outputs/mom6-panan/panant-01-hycom1-v13 | ||
|
||
- metadata_yaml: /g/data/ik11/outputs/mom6-panan/panant-01-zstar-v13/metadata.yaml | ||
path: | ||
- /g/data/ik11/outputs/mom6-panan/panant-01-zstar-v13 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
Generated using the following Python on NCI ARE: | ||
|
||
``` | ||
import netCDF4 | ||
import glob | ||
import os | ||
|
||
srcdir = "/g/data/ik11/outputs/mom6-om4-025/OM4_025.JRA_RYF/" | ||
destdir = "/home/120/mcw120/scratch/" | ||
|
||
targets = ["output000", "output001"] | ||
target_glob = "190*.nc" | ||
|
||
for target in targets: | ||
srcglob = os.path.join(srcdir, target, target_glob) | ||
srcfns = [os.path.basename(_) for _ in glob.glob(srcglob)] | ||
for srcfn in srcfns: | ||
with netCDF4.Dataset(os.path.join(srcdir, target, srcfn), "r") as old, netCDF4.Dataset(os.path.join(destdir, target, srcfn), "w") as new: | ||
print(old.filepath()) | ||
print(new.filepath()) | ||
new.setncattr("file_format", old.file_format) | ||
new.setncatts(old.__dict__) | ||
# new.set_fill_off() | ||
for name, dimension in old.dimensions.items(): | ||
if name == "nv": | ||
dimn = 2 | ||
elif not dimension.isunlimited(): | ||
dimn = 1 | ||
else: | ||
dimn = dimension.size | ||
new.createDimension(name, dimn) | ||
for name, variable in old.variables.items(): | ||
x = new.createVariable(name, variable.datatype, variable.dimensions) | ||
new[name].setncatts(old[name].__dict__) | ||
if name == "time" or name == "nv": # Need time-related data only | ||
new[name][:] = old[name][:] | ||
print(">> Complete!") | ||
print("--") | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
CPATH: /apps/openmpi/4.0.3/include | ||
CPATH_modshare: /apps/openmpi/4.0.3/include:1 | ||
CPLUS_INCLUDE_PATH: /apps/openmpi/4.0.3/include | ||
CPLUS_INCLUDE_PATH_modshare: /apps/openmpi/4.0.3/include:1 | ||
C_INCLUDE_PATH: /apps/openmpi/4.0.3/include | ||
C_INCLUDE_PATH_modshare: /apps/openmpi/4.0.3/include:1 | ||
ENVIRONMENT: BATCH | ||
FPATH: /apps/openmpi/4.0.3/include | ||
FPATH_modshare: /apps/openmpi/4.0.3/include:1 | ||
GIT_CONFIG_NOGLOBAL: 'yes' | ||
HOME: /home/157/amh157 | ||
LC_CTYPE: C.UTF-8 | ||
LD_LIBRARY_PATH: /apps/openmpi/4.0.3/lib:/apps/openmpi/4.0.3/lib/profilers | ||
LD_LIBRARY_PATH_modshare: /apps/openmpi/4.0.3/lib:1:/apps/openmpi/4.0.3/lib/profilers:1 | ||
LD_RUN_PATH: /apps/openmpi/4.0.3/lib:/apps/openmpi/4.0.3/lib/profilers | ||
LD_RUN_PATH_modshare: /apps/openmpi/4.0.3/lib:1:/apps/openmpi/4.0.3/lib/profilers:1 | ||
LIBRARY_PATH: /apps/openmpi/4.0.3/lib:/apps/openmpi/4.0.3/lib/profilers | ||
LIBRARY_PATH_modshare: /apps/openmpi/4.0.3/lib:1:/apps/openmpi/4.0.3/lib/profilers:1 | ||
LOADEDMODULES: pbs:openmpi/4.0.3 | ||
LOADEDMODULES_modshare: pbs:1:openmpi/4.0.3:1 | ||
LOGNAME: amh157 | ||
MANPATH: /apps/openmpi/4.0.3/share/man:/opt/pbs/default/share/man | ||
MANPATH_modshare: /apps/openmpi/4.0.3/share/man:1:/opt/pbs/default/share/man:1 | ||
MODULEPATH: /g/data3/hh5/public/modules:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles | ||
MODULESHOME: /opt/Modules/v4.3.0 | ||
MODULES_CMD: /opt/Modules/v4.3.0/libexec/modulecmd.tcl | ||
MODULES_LMCONFLICT: openmpi/4.0.3&mpi&lam&mpich&openmpi&intel-mpi&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars | ||
MODULES_LMCONFLICT_modshare: openmpi/4.0.3&mpi&lam&mpich&openmpi&intel-mpi&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1 | ||
MODULE_VERSION: v4.3.0 | ||
MODULE_VERSION_STACK: v4.3.0 | ||
MXM_LOG_FILE: /dev/null | ||
MXM_LOG_LEVEL: ERROR | ||
NCPUS: '48' | ||
OMPI_BASE: /apps/openmpi/4.0.3 | ||
OMPI_MCA_orte_tmpdir_base: /jobfs/28342543.gadi-pbs | ||
OMPI_ROOT: /apps/openmpi/4.0.3 | ||
OMPI_VERSION: 4.0.3 | ||
OMP_NUM_THREADS: '48' | ||
OPENMPI_BASE: /apps/openmpi/4.0.3 | ||
OPENMPI_ROOT: /apps/openmpi/4.0.3 | ||
OPENMPI_VERSION: 4.0.3 | ||
PATH: /apps/openmpi/wrapper/fortran:/apps/openmpi/wrapper:/apps/openmpi/4.0.3/bin:/bin:/usr/bin:/opt/pbs/default/bin | ||
PATH_modshare: /apps/openmpi/4.0.3/bin:1:/bin:1:/apps/openmpi/wrapper/fortran:1:/usr/bin:1:/apps/openmpi/wrapper:1:/opt/pbs/default/bin:1 | ||
PAYU_PATH: /g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.07/bin | ||
PBS_ENVIRONMENT: PBS_BATCH | ||
PBS_JOBCOOKIE: 7366EFF670A4AAA3664067773F6676EF | ||
PBS_JOBDIR: /home/157/amh157 | ||
PBS_JOBFS: /jobfs/28342543.gadi-pbs | ||
PBS_JOBID: 28342543.gadi-pbs | ||
PBS_JOBNAME: OM4_025.JRA | ||
PBS_MOMPORT: '15003' | ||
PBS_NCI_FS_GDATA1: '0' | ||
PBS_NCI_FS_GDATA1A: '0' | ||
PBS_NCI_FS_GDATA1B: '0' | ||
PBS_NCI_FS_GDATA2: '0' | ||
PBS_NCI_FS_GDATA3: '0' | ||
PBS_NCI_FS_GDATA4: '0' | ||
PBS_NCI_HT: '0' | ||
PBS_NCI_IMAGE: '' | ||
PBS_NCI_JOBFS: 10gb | ||
PBS_NCI_LAUNCH_COMPATIBILITY: '0' | ||
PBS_NCI_NCPUS_PER_NODE: '48' | ||
PBS_NCI_NCPUS_PER_NUMA: '12' | ||
PBS_NCI_NUMA_PER_NODE: '4' | ||
PBS_NCI_STORAGE: scratch/x77+gdata/ua8+gdata/hh5+gdata/x77 | ||
PBS_NCI_WD: '1' | ||
PBS_NCPUS: '1776' | ||
PBS_NGPUS: '0' | ||
PBS_NNODES: '37' | ||
PBS_NODEFILE: /local/spool/pbs/aux/28342543.gadi-pbs | ||
PBS_NODENUM: '0' | ||
PBS_O_HOME: /home/157/amh157 | ||
PBS_O_HOST: gadi-login-02.gadi.nci.org.au | ||
PBS_O_LANG: en_AU.UTF-8 | ||
PBS_O_LOGNAME: amh157 | ||
PBS_O_MAIL: /var/spool/mail/amh157 | ||
PBS_O_PATH: /apps/ncview/2.1.7/bin:/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.07/bin:/g/data3/hh5/public/apps/miniconda3/condabin:/home/157/amh157/.local/bin:/home/157/amh157/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/default/bin | ||
PBS_O_QUEUE: normal | ||
PBS_O_SHELL: /bin/bash | ||
PBS_O_SYSTEM: Linux | ||
PBS_O_TZ: :/etc/localtime | ||
PBS_O_WORKDIR: /home/157/amh157/mom6/OM4_025.JRA | ||
PBS_QUEUE: normal-exec | ||
PBS_TASKNUM: '1' | ||
PBS_VMEM: '7627861917696' | ||
PROJECT: x77 | ||
SHELL: /opt/bin/nfsh | ||
TMPDIR: /jobfs/28342543.gadi-pbs | ||
USER: amh157 | ||
VT_MAX_FLUSHES: '0' | ||
VT_PFORM_LDIR: /jobfs/28342543.gadi-pbs | ||
_LMFILES_: /opt/Modules/modulefiles/pbs:/apps/Modules/modulefiles/openmpi/4.0.3 | ||
_LMFILES__modshare: /apps/Modules/modulefiles/openmpi/4.0.3:1:/opt/Modules/modulefiles/pbs:1 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Codecov is complaining that this line isn't tested - tbh I think it's unimportant.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm assuming the equivalent lines aren't tested in the other
Builders
, but I might look into that.