Don't skip files based on filename

[aekiss]

This skip based on filename is incorrect, because the upstream repository doesn't update the filename of complete year files, even though it continues to update the data in the file as the finalised reanalysis comes in. So if the script is run in January it will download unfinalised data for December of the previous year, and later re-runs of the script won't update the file because the filename is unchanged and therefore skipped.

JRA55-do-1-5-0-1/update.py

Lines 25 to 26 in 9452f99

	if os.path.exists(outfn):
	print('--- skipping', outfn, '(already exists)')

[aekiss]

This comparison between all the full-year .nc files

/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/*/*/*/*/*/*1231*.nc
/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror-2024-03-14/*/*/*/*/*/*1231*.nc

shows that this bug has only affected the following files in mirror:

/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/atmos/3hrPt/huss/gr/latest/huss_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202301010000-202312312100.nc
/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/atmos/3hrPt/tas/gr/latest/tas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202301010000-202312312100.nc
/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/atmos/3hrPt/uas/gr/latest/uas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202301010000-202312312100.nc
/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/atmos/3hrPt/vas/gr/latest/vas_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gr_202301010000-202312312100.nc
/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/ocean/day/tos/gn/latest/tos_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gn_20230101-20231231.nc
/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror/seaIce/day/siconc/gn/latest/siconc_input4MIPs_atmosphericState_OMIP_MRI-JRA55-do-1-5-0-1_gn_20230101-20231231.nc

with the first differences occurring at 2023-12-10T15:00:00.

So we got lucky, and only Dec 2023 was affected, not the previous years.

Even better, 01deg_jra55v140_iaf_cycle4_jra55v150_extension is OK, as it was only run up to 2023-10-01 so far.

😅

[aekiss]

The data in mirror-2024-03-14 is finalised up to the end of 2023 but (surprisingly) not finalised for 2024, so don't use mirror-2024-03-14 for 2024.

[rmholmes]

@aekiss I just reran my 025deg_jra55_iaf_era5comparison run for 2023 with mirror replaced with mirror-2024-03-14 and, at least for monthly average SST, there is no difference between the two runs.

Compare /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_era5comparison/output043/ (mirror) with /scratch/v45/rmh561/access-om2/archive/025deg_jra55v150_iaf/output043 (mirror-2024-03-14). I differenced the SST in /scratch/v45/rmh561/access-om2/archive/025deg_jra55v150_iaf/SST_diff.nc.

This is surprising to me. Did I do something wrong maybe?

[aekiss]

Hm, I'm seeing differences, eg between these

/g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_era5comparison/output043/ice/OUTPUT/iceh.2023-12.nc
/scratch/v45/rmh561/access-om2/archive/025deg_jra55v150_iaf/output043/ice/OUTPUT/iceh.2023-12.nc

/g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_era5comparison/output043/ocean/ocean-2d-bottom_temp-1-daily-max-ym_2023_01.nc
/scratch/v45/rmh561/access-om2/archive/025deg_jra55v150_iaf/output043/ocean/ocean-2d-bottom_temp-1-daily-max-ym_2023_01.nc

/g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_era5comparison/output043/ocean/ocean-2d-frazil_3d_int_z-1-monthly-mean-ym_2023_01.nc
/scratch/v45/rmh561/access-om2/archive/025deg_jra55v150_iaf/output043/ocean/ocean-2d-frazil_3d_int_z-1-monthly-mean-ym_2023_01.nc

and more

[aekiss]

/scratch/v45/rmh561/access-om2/archive/025deg_jra55v150_iaf/SST_diff.nc isn't zero - ncview says "displayed range: 0.141724 to 0.101624 K" and displays a difference in Dec. Difference appears to be zero in prior months, as expected.

[rmholmes]

Ah sorry, yes you're right. I had my ncview limits wrong.

All good. I'll replace the run in ik11 with this one.

This is the only run I know of which uses mirror for December 2023. So happy for you to replace the files in mirror with the fixed ones, if that is what you want to do.

[aekiss]

/g/data/ik11/inputs/JRA-55/JRA55-do-1-5-0-1/mirror-2024-04-11 contains finalised data for the entire dataset (including Jan 2024), so this is the dataset to use.

I've renamed mirror to mirror-DO-NOT-USE so it's still available for provenance in case anyone used it.