NERSC GPU Hackathon 2021

For the hackathon we will be using (in order):

• Perlmutter Phase 1
• Ascent
• deep bayou

Steve Brandt and Roland Haas worked a bit to see how to compile on these systems and came up with two methods. One "spack" uses pre-compiled libraries provided by a spack installation that Steve created and he other "ExternalLibraries" uses the tarballs in the ExternalLibraries to compiled code.

These instructions assume that you are familiar with Cactus and Simfactory. You may want to consult Getting Started if you are not.

These instructions are currently work-in-progress, please report any issues that you find via the shared GRHydroX team chat (or a ticket here). Please do not use direct messages to Steve or Roland (they may get lost) for these.

Perlmutter Phase 1

Spack

TODO: describe this properly

This uses a plain Cactus build. First check out the version of the code you would like, eg:

Log in to Perlmutter phase 1:

ssh [email protected]

where $USER is your username on NERSC. Then

./GetComponents https://bitbucket.org/eschnett/cactusamrex/raw/master/azure-pipelines/carpetx-cuda.th

and set up the modules required:

module load gcc/9.3.0
source foo/sbrant/spack.sh # this may still fail right now

Then compile Cactus:

cd Cactus
make sim-config options=/global/u2/s/sbrandt/spack-cuda.cfg THORNLIST=thornlists/carpetx-cuda.th
make -j8 sim

ExternalLibraries

This mostly follows Getting Started using the same type of thorns.

Log in to Perlmutter phase 1:

ssh [email protected]

where $USER is your username on NERSC. Then

./GetComponents --root Cactus --parallel https://bitbucket.org/eschnett/cactusamrex/raw/d659fbcb2593ab4816040c3b0b16203bc3c22ffe/manifest/carpetx.th

Switch simfactory the rhaas/perlmutter branch, switch cactusamrex to rhaas/cuda:

cd repos/simfactory2
git checkout rhaas/perlmutter
cd ../repos/cactusamrex
git checkout rhaas/cuda
cd ../../

Set up user name etc. for perlmutter:

echo perlmutter-p1 >~/.hostname
./simfactory/bin/sim setup-silent

Compile:

./simfactory/bin/sim build --thornlist thornlists/carpetx.th

Run a test:

./simfactory/bin/sim submit --allocation m3374 --cores 128 --walltime 0:5:0 --parfile repos/cactusamrex/WaveToyGPU/par/planewave-gpu.par planewave-gpu-01

Ascent / Summit

Spack

TODO: add this

ExternalLibraries

These instructions were tested on Summit since I (Roland) do not yet have access to Ascent, which is the open, training system using Summit hardware.

Log in:

ssh [email protected]

where $USER is your username on NERSC. Then change to scratch (I got build failures when I tried to build in $HOME which uses NFS):

cd /gpfs/alpine/*/scratch/$USER

Then get the code:

./GetComponents --root Cactus --parallel https://bitbucket.org/eschnett/cactusamrex/raw/d659fbcb2593ab4816040c3b0b16203bc3c22ffe/manifest/carpetx.th

Switch simfactory the rhaas/summit-gpu branch, switch cactusamrex to rhaas/cuda:

cd repos/simfactory2
git checkout rhaas/summit-gpu
cd ../repos/cactusamrex
git checkout rhaas/cuda
cd ../../

Change summit's machine description to cause the updated files by editing simfactory/etc/defs.local.ini and adding a section for summit:

[summit]
# you may have to adjust the allocation
allocation = GEN170
enabled-thorns = <<EOF
ExternalLibraries/CUDA
CarpetX/WaveToyGPU
EOF
optionlist = summit-gpu.cfg
# load some additional modules
envsetup = <<EOF
    [ -n "$LMOD_CMD" ] || source /etc/profile
    module unload boost || true
    module unload spectrum-mpi || true
    module unload gcc || true
    module unload xl || true
    module unload cuda || true
    module load gcc/9.3.0 &&
    module load spectrum-mpi/10.4.0.3-20210112 &&
    module load cuda/11.3.1 &&
    module load gsl/2.5 &&
    module load hdf5/1.10.7 &&
    module load boost/1.77.0 &&
    module load openblas/0.3.17 &&
    module load fftw/3.3.8 &&
    module load curl/7.79.0 &&
    module load hwloc/2.5.0 &&
    module load hypre/2.22.0-cpu &&
    module load papi/6.0.0.1 &&
    module load perl/5.30.1 &&
    module load petsc/3.15.4-no_cuda &&
    module load zlib/1.2.11 &&
    module load cmake/3.21.3
EOF

Set up user name etc for Summit:

./simfactory/bin/sim setup-silent

Compile:

./simfactory/bin/sim build --thornlist thornlists/carpetx.th