Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update doc files #118

Merged
merged 1 commit into from
Sep 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions doc/MOM_input
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ NJGLOBAL = 300 !
! domain. With STATIC_MEMORY_ this is set in MOM_memory.h at compile time.

! === module MOM_fixed_initialization ===
INPUTDIR = "./input/" ! default = "."
INPUTDIR = "./INPUT/" ! default = "."
! The directory in which input files are found.

! === module MOM_grid_init ===
Expand Down Expand Up @@ -141,6 +141,12 @@ DTFREEZE_DP = -7.75E-08 ! [deg C Pa-1] default = 0.0
! When TFREEZE_FORM=LINEAR, this is the derivative of the freezing potential
! temperature with pressure.

EPS_OMESH = 1e-13
! "default = 0.0001
! An float which sets the allowable error (in degrees) between
! grid angle defined in the ESMF mesh file used by CMEPS
! and the ocean_hgrid file used by mom

! === module MOM_tracer_flow_control ===
USE_IDEAL_AGE_TRACER = True ! [Boolean] default = False
! If true, use the ideal_age_example tracer package.
Expand Down Expand Up @@ -201,6 +207,9 @@ Z_INIT_FILE_PTEMP_VAR = "temp" ! default = "ptemp"
! The name of the potential temperature variable in TEMP_Z_INIT_FILE.
Z_INIT_ALE_REMAPPING = True ! [Boolean] default = False
! If True, then remap straight to model coordinate from file.
Z_INIT_REMAP_GENERAL = True ! [Boolean] default = False
! If false, only initializes to z* coordinates. If true, allows initialization
! directly to general coordinates.
TEMP_SALT_INIT_VERTICAL_REMAP_ONLY = True ! [Boolean] default = False
! If true, initial conditions are on the model horizontal grid. Extrapolation
! over missing ocean values is done using an ICE-9 procedure with vertical ALE
Expand Down Expand Up @@ -620,8 +629,6 @@ SALT_RESTORE_FILE = "salt_sfc_restore.nc" ! default = "salt_restore.nc"
SRESTORE_AS_SFLUX = True ! [Boolean] default = False
! If true, the restoring of salinity is applied as a salt flux instead of as a
! freshwater flux.
MAX_DELTA_SRESTORE = 0.5 ! [PSU or g kg-1] default = 999.0
! The maximum salinity difference used in restoring terms.
GUST_CONST = 0.02 ! [Pa] default = 0.0
! The background gustiness in the winds.

Expand Down
1 change: 1 addition & 0 deletions doc/README_doc.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ The files in here should be the minimum required to support code search from lin
## Update history
1. Updated Mon 12 Feb 2024 14:22:51 AEDT from branch [1deg_jra55do_ryf](https://github.com/ACCESS-NRI/access-om3-configs/tree/1deg_jra55do_ryf) commit [a6829f9](https://github.com/ACCESS-NRI/access-om3-configs/tree/a6829f9)
1. Updated Wed 27 Mar 2024 12:59:33 AEDT from branch [1deg_jra55do_ryf](https://github.com/ACCESS-NRI/access-om3-configs/tree/1deg_jra55do_ryf) commit [c2585c7](https://github.com/ACCESS-NRI/access-om3-configs/tree/c2585c7)
1. Updated Mon 2 Sep 2024 16:14:36 AEST from branch [1deg_jra55do_ryf](https://github.com/ACCESS-NRI/access-om3-configs/tree/1deg_jra55do_ryf) commit [2bc6107](https://github.com/ACCESS-NRI/access-om3-configs/tree/2bc6107)
47 changes: 28 additions & 19 deletions doc/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,41 +2,50 @@

# If submitting to a different project to your default, uncomment line below
# and change project code as appropriate; also set shortpath below
project: x77
# project: x77

# Force payu to always find, and save, files in this scratch project directory
# (you may need to add the corresponding PBS -l storage flag in sync_data.sh)
shortpath: /scratch/v45
# shortpath: /scratch/v45

queue: normal
ncpus: 48
ncpus: 240
jobfs: 10GB
mem: 192GB
mem: 960GB

walltime: 01:00:00
jobname: 1deg_jra55do_ryf

model: access-om3

exe: /g/data/ik11/spack/0.20.1/opt/linux-rocky8-cascadelake/intel-2021.6.0/access-om3-main-lxbeca7/bin/access-om3-MOM6-CICE6
input:
- /g/data/ik11/inputs/access-om3/0.x.0/1deg/share # shared grids and topography
- /g/data/ik11/inputs/access-om3/0.x.0/1deg/mom # grids, ICs etc
- /g/data/ik11/inputs/access-om3/0.x.0/1deg/cice # grids, ICs etc
- /g/data/ik11/inputs/access-om3/0.x.0/share/meshes/JRA55do-ESMFmesh.nc # mesh for JRA55-do stream
- /g/data/ik11/inputs/JRA-55/RYF/v1-4 # datm and drof JRA55-do streams data
exe: access-om3-MOM6-CICE6
input:
- /g/data/vk83/experiments/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-ESMFmesh.nc
- /g/data/vk83/experiments/inputs/access-om3/share/meshes/global.1deg/2024.01.25/access-om2-1deg-nomask-ESMFmesh.nc
- /g/data/vk83/experiments/inputs/access-om3/share/meshes/share/2024.01.25/JRA55do-ESMFmesh.nc
- /g/data/vk83/experiments/inputs/access-om3/share/grids/global.1deg/2020.10.22/topog.nc
- /g/data/vk83/experiments/inputs/access-om3/mom/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
- /g/data/vk83/experiments/inputs/access-om3/mom/grids/vertical/global.1deg/2023.07.28/ocean_vgrid.nc
- /g/data/vk83/experiments/inputs/access-om3/mom/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc
- /g/data/vk83/experiments/inputs/access-om3/mom/surface_salt_restoring/global.1deg/2020.05.30/salt_sfc_restore.nc
- /g/data/vk83/experiments/inputs/access-om3/cice/grids/global.1deg/2024.05.14/grid.nc
- /g/data/vk83/experiments/inputs/access-om3/cice/grids/global.1deg/2024.05.14/kmt.nc
- /g/data/vk83/experiments/inputs/access-om3/cice/initial_conditions/global.1deg/2023.07.28/iced.1900-01-01-10800.nc
- /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data

collate: false
runlog: false
metadata:
enable: false

userscripts:
setup: ./setup_cice_restarts.sh
archive: ./archive_cice_restarts.sh
setup: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/setup.sh
archive: /usr/bin/bash /g/data/vk83/apps/om3-scripts/payu_config/archive.sh

modules:
use:
- /g/data/ik11/spack/0.20.1/modules/access-om3/0.x.0/linux-rocky8-cascadelake
load:
- intel-compiler/2021.6.0
- openmpi/4.1.4
- parallelio/2.5.10
use:
- /g/data/ik11/spack/0.21.2/modules/access-om3/0.x.0/linux-rocky8-cascadelake
- /g/data/hh5/public/modules
load:
- access-om3/63ed4a4777aeac0c62720ddbe7f85e4ed16f82eb_main
- conda/analysis3-24.01
39 changes: 28 additions & 11 deletions doc/ice_in
Original file line number Diff line number Diff line change
@@ -1,29 +1,32 @@
&setup_nml
bfbflag = "off"
conserv_check = .false.
debug_forcing = .true.
debug_model = .true.
diagfreq = 960
dumpfreq = "y"
dump_last = .true.
histfreq = "d", "m", "x", "x", "x"
hist_suffix = "", "m", "x", "x", "x" !limited to one character due to https://github.com/COSIMA/access-om3/issues/201
hist_time_axis = "middle"
history_deflate = 1
history_chunksize = 180, 150
history_precision = 8
ice_ic = "./input/iced.1900-01-01-10800.nc"
restart_deflate = 1
ice_ic = "./INPUT/iced.1900-01-01-10800.nc"
lcdf64 = .false.
npt = 35040
pointer_file = './rpointer.ice'
print_global = .false.
/
&grid_nml
bathymetry_file = "./input/topog.nc"
bathymetry_file = "./INPUT/topog.nc"
grid_atm = "A"
grid_file = "./input/grid.nc"
grid_file = "./INPUT/grid.nc"
grid_format = "nc"
grid_ice = "B"
grid_ocn = "A"
grid_type = "tripole"
kcatbound = 0
kmt_file = "./input/kmt.nc"
kmt_file = "./INPUT/kmt.nc"
nblyr = 1
ncat = 5
nfsd = 1
Expand Down Expand Up @@ -65,18 +68,18 @@
ustar_min = 0.0005
/
&domain_nml
block_size_x = 16
block_size_y = 15
block_size_x = 15
block_size_y = 20
Comment on lines +71 to +72

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As was discussed in COSIMA/access-om3#214 (comment) and cice docs, It is suggested that max_blocks be set in the range of 3-8 blocks per processor.

Suggested change
block_size_x = 15
block_size_y = 20
block_size_x = 30
block_size_y = 40

With the updated block_size_x and block_size_y, max_blocks is determined to be 8, which suits the recommendation from the cice docs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the slender distributions block_size_x should equal nx_global/nprocs i.e. 360/24 = 15. Which is why these block sizes are set as they are at the moment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anton-seaice. So for slenderX1, the block size in the y direction should be 300 instead of 20, right? This is because there is only one processor in the y direction, and it needs to handle all 300 grid points.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could try that. That would mean there is 1 block per processor, which means you couldn't eliminate any blocks which are entirely land. I am not sure how it could impact inter-process communication ? I suspect the performance will be about the same as the current configuration ?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont get it. If that’s the case, why does block_size_x equal nx_global / nprocs, but block_size_y is calculated differently?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its just kind of by definition, you can have multiple blocks in Y but not X. Each color is a different processor:

Screenshot 2024-09-03 at 1 36 33 PM

distribution_type = "cartesian"
distribution_wght = "latitude"
maskhalo_bound = .true.
maskhalo_dyn = .true.
maskhalo_remap = .true.
max_blocks = 10
max_blocks = 15

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
max_blocks = 15
max_blocks = -1

In the latest 025deg_jra55do_ryf, max_blocks has been set to -1 to allow the code to automatically determine the optimal value.

max_blocks = -1

I can quickly update the 1deg config.

ns_boundary_type = "tripole"
nx_global = 360
ny_global = 300
processor_shape = "slenderX2"
processor_shape = "slenderX1"
/
&ice_prescribed_nml
/
Expand Down Expand Up @@ -123,12 +126,26 @@
f_fsurfn_ai = "m"
f_hi = "md"
f_hs = "md"
f_sifb = "md"
f_snoice = "md"
f_uvel = "md" , f_vvel = "md"
f_vicen = "m"
!-----------------------------------
! These fields are on by default (in ice_history_shared.F90) but lets turn them off
!-----------------------------------
f_tlon = .false. , f_tlat = .false.
f_ulon = .false. , f_ulat = .false.
f_nlon = .false. , f_nlat = .false.
f_elon = .false. , f_elat = .false.
f_tmask = .false. , f_umask = .false.
f_nmask = .false. , f_emask = .false.
f_tarea = .false. , f_uarea = .false.
f_narea = .false. , f_earea = .false.
f_dxt = .false. , f_dyt = .false.
f_dxu = .false. , f_dyu = .false.
f_dxe = .false. , f_dye = .false.
f_dxn = .false. , f_dyn = .false.
f_HTN = .false. , f_HTE = .false.
f_albpnd = 'x'
f_atmdir = 'x' , f_atmspd = 'x'
f_coszen = 'x'
Expand Down Expand Up @@ -157,4 +174,4 @@
f_taubx = 'x' , f_tauby = 'x'
f_Tref = 'x', f_Qref = 'x'
f_uocn = 'x' , f_vocn = 'x'
/
/
Loading