Levante Experiences

[pgierz]

Please post any info and updates about using any of our models on the new DKRZ Levante system here!

[pgierz]

AWIESM 2.1

I will edit this comment here over time to provide updates.

I have made a branch based on feat/awiesm-final and merged (most) of the feat/levante in. There were some conflicts, in particular with fesom-2.1 that we will need to examine by hand.

So far, compile tests:

• oasis: no problem
• fesom: no problem
• echam: Fails

checking whether MPI_Send accepts const void * as first argument... yes
checking type of MPI_Aint... long
checking for mpirun... /sw/spack-levante/openmpi-4.1.2-yfwe6t/bin/mpirun
checking if /sw/spack-levante/openmpi-4.1.2-yfwe6t/bin/mpirun works... no
configure: error: in `/work/ab0246/a270077/SciComp/Model_Support/awiesm_porting/model_codes/awiesm-2.1/echam-6.3.05p2/yaxt':
configure: error: failed to run MPI programs
See `config.log' for more details
configure: error: ./configure failed for yaxt

EDIT 16.06.2022

AWIESM 2.1 now builds and runs correctly on Levante.

[seb-wahl]

FOCI

I made a lot of changes (cleanup, ...) to the levante.yaml to be able to include FOCI. I structured it in the same way as we did for e.g. glogin.yaml. This stuff is not yet committed.
Current status: FOCI (ECHAM6 + OASIS + NEMO) compiles (intelmpi) but crashes right upon startup with error messages that don't give me a hint what's going on. I contacted DKRZ and got the following reply:

Dear Sebastian,

there is currently one working setup known to me, but some testing by Kalle (in CC) and Michael Botzet from MPI-M is still needed before it can be released. Please, contact Kalle for more details. In case you are using Intel MPI instead of Open MPI, that is known to be problematic in some cases, since some patches from our side are still needed.

Kind regards,
Irina Fast

28.03.2022 09:15 Wahl, Sebastian wrote:

Dear Support

Do you have (or know someone) who has run ECHAM6 and/or MPIESM (or any other
coupled setup using OASIS) on levante? If yes would you or someone you know be
willing to share their settings (compile flags, runtime environment variables,
...) and experiences?

While compilation is fine I’m currently cannot run our coupled setup FOCI (NEMO
ocean with ECHAM6 atmosphere coupled via OASIS). It crashes right at the beginning
without any useful error message.

Sebastian

[seb-wahl]

Just pushed my changes to feat/levante. FOCI is still not running but that might be due to the issues as MPI mentioned above.

[mandresm]

AWICM3

IO library problems

• module's ECCODES compiled without OMP_THREADS option. Submitted a support request but no answer yet.
• the solution to that will be to use Sebastian's IO libs

Compilation

• compiles all models with Open MPI with dkrz modules
  • problems in the linking of ECCODES
• TODO: compile with @seb-wahl IO libraries. @seb-wahl can you provide a path for those?

Running

• Running fails due to the problems with ECCODES linking

[seb-wahl]

Sorry I missed this one (I promised to do this on Monday). I just recompiled eccodes with
-DENABLE_ECCODES_OMP_THREADS=ON. This version is now available at /work/bb0519/HPC_libraries/intel2022.0.1_impi2021.5.0_20220318/

[mandresm]

Thanks a lot Sebastian!

[seb-wahl]

After helpful input from Enrico from DKRZ (worked with him on the workshop) I was to run FOCI with OpenMP at top speed. As we have the cmake compile for ECHAM6 here are the compile flags (the most important ones are -march and -mtune:
-O3 -fp-model source -qoverride-limits -assume realloc_lhs -align array64byte -no-prec-sqrt -no-prec-div -fast-transcendentals -m64 -march=core-avx2 -mtune=core-avx2 -fma -ip -pc64
I used the following recommended CPU layout for ECHAM6:

&parctl
    nproca = 32
    nprocb = 24
/
&runctl
   nproma = 8
/

[seb-wahl]

FOCIOIFS runs on levante at 35 to 45min / year without any special tuning or paying attention whether the CPU distribution to ocean and atmosphere are in balance. I'll stop here for now as we don't have an active project at the moment where we need FOCIOIFS to run fast on levante. I'm curious to learn how OIFS standalone or the AWI models with OIFS perform.
Changes have just been pushed to feat/levante, see https://github.com/esm-tools/esm_tools/tree/feat/levante

[mandresm]

I've being trying to run yesterday AWI-CM3 with very little success, but I just got a running simulation for the first now some minutes ago, so I hope I can add some performance details by the end of the day. @seb-wahl , the branch where things work for you is geomar right?

For AWI-CM1, I managed to compile ECHAM-6.3.04p1 with the new flags, but FESOM is still a problem due to the missing netcdf_c++ library (I'm gonna contact DKRZ for requesting that).

[denizural]

I had the same problem on Mistral. I built C++ NetCDF manually on my home account and chmod 777 for everyone and informed DKRZ about that so that they could update their out of date library. Unfortunately, noone updated that 😞

[mandresm]

Thanks for the info. I have submitted a ticket to DKRZ support asking for NetCDF C++ module. If they provide it great, if not, maybe we can have tomorrow a session together trying to get it compiled for Levante.

[seb-wahl]

No, I used the feat/levante branch. I used the new flags for ECHAM6 and -march=core-avx2 -mtune=core-avx2 of them for NEMO and OASIS).

[seb-wahl]

...and I used the system libraries, except for ECCODES for OpenIFS where I used the version that Jan compiles, it's all in my latest commits.

[mandresm]

Some updates on AWICM1: DKRZ has put together the modules for the NetCDF C++ for the different compilers and mpis. However, AWICM1 still does not compile because it cannot find netcdf include file. I contacted DKRZ to notify them that, it seems that the include file netcdf is missing in the include folder of the NetCDF C++ library.

In parallel I managed to compile NetCDF C++ with the help of some notes from @denizural and the compilation of AWICM1 with that was successful.

[Mccino]

Dear supports,
I got one problem on Levante while installing model.
When I started to install a model, for example, using: esm_master install-besom-2.0, I got the message:

WARNING:root:The yaml file for this computer (levante3.atos.local) could not be determined!
WARNING:root:Continuing with generic settings...

And then the errors are:

ERROR: Unable to locate a modulefile for 'something'
subprocess.CalledProcessError: Command '['./comp-fesom-2.0_script.sh']' returned non-zero exit status 1.

Did I make something wrong on Levante or do you have any experience with how I can solve this?

Thanks.

[pgierz]

Dear @Mccino, the levante support feature will be merged in later this afternoon. @mandresm is handling user support today and will get back to you.

[denizural]

Fesom 1.4:

Fesom 1.4 run from @vkolatschek

Mistral	Levante
Mesh: GLOB_rmstep	Mesh: GLOB_rmstep
nod2d: 830 305	nod2d: 830 305
nod3d: 24 185 969	nod3d: 24 185 969
~0.19 sec/step	~0.86 sec/step
nproc: 1920	nproc: 1920

Dima only reported a 10-20% decrease in computation time.

[vkolatschek]

Sorry! It should work now.

[mandresm]

Nope :S I cannot access them:

cd /home/a/a270147/fesom-1.4_no_cavity/
-bash: cd: /home/a/a270147/fesom-1.4_no_cavity/: Permission denied

[vkolatschek]

Try again? I did it recursive this time.

[mandresm]

I've checked your FESOM-1.4 code, and although the branch is master it does not have the latest changes in the fesom_cpl/CMakeLists.txt necessary for running faster at Levante. Please, make sure you have your branch up-to-date (or at least the cmake files), recompile and try again.

[vkolatschek]

Thank you! That solved this problem!

Mistral	Levante (before)	Levante (after)
Mesh: GLOB_rmstep	Mesh: GLOB_rmstep	Mesh: GLOB_rmstep
nod2d: 830 305	nod2d: 830 305	nod2d: 830 305
nod3d: 24 185 969	nod3d: 24 185 969	nod3d: 24 185 969
~0.19 sec/step	~0.86 sec/step	~0.16 sec/step
nproc: 1920	nproc: 1920	nproc: 1920