diff --git a/configs/README.md b/configs/README.md index 201fa0a..727b729 100644 --- a/configs/README.md +++ b/configs/README.md @@ -16,10 +16,10 @@ The beatnik spack package should enforce its build requirements appropriately; w ### Current list of tested systems targets and suggested installation method -We have tested beatnik installation on the following systems via either spack install with the provided spack install flags or a spack environment. An example run script is also provided for each of these systems in the appropriate subdirectory. - * University of New Mexico - These systems simply use `spack install` as the UNM machines have a full global spack packages.yaml already set up. Because thse systems use spack for package maintenance, we recommend adding the provided `upstreams.yaml` file in the `unm` directory to the `.spack` directory in your home directory. - * Hopper V100/A100 GPU cluster system - `spack install beatnik +cuda cuda_arch=80 %gcc ^cuda@11` (or `cuda_arch=70` for the V100 nodes); ^cuda@11 is needed to avoid using CUDA 12 which the UNM cuda drivers aren't updated to support as of 10/1/23. Only tested with gcc as the compiler. - * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is sufficient +We have tested the beatnik installation on the following systems via either spack install with the provided spack install flags or a spack environment. An example run script is also provided for each of these systems in the appropriate subdirectory. + * University of New Mexico - UNM systems have a full global spack packages.yaml already set up, and so it is often possible to use `spack install` with the appropriate compiler and other flags to install beatnik on these systems. We recommend adding the provided `upstreams.yaml` file in the `unm` directory to the `.spack` directory in your home directory. + * Hopper V100/A100 GPU cluster system - Build using the environment specification in `unm/hopper/spack.yaml`. It is sometimes possible to simply use `spack install beatnik +cuda cuda_arch=80 %gcc ^cuda@11` (or `cuda_arch=70` for the V100 nodes), but this may not always work depending on how spack chooses to concretize the spack installation. + * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is generally sufficient. * Lawrence Livermore National Laboratory - These systems need a spack environment (provided) to set up compilers and external packages to use spack effectively on these systems. Use `spack env create` with the provided spack.yaml to build beatnik in an environment on these systems. Simple test run scripts are also provided. * Lassen V100 GPU system - Build using the environment specification in llnl/lassen/spack.yaml. Other compilers besides gcc untested. Note that cuda-aware MPI support is broken in spectrum MPI with cuda versions later than 11.2; the spack configuration avoids this, but if you're building beatnik by hand on Lassen or another IBM CORAL system, be aware of this limitation. * Tioga MX250X GPU system - Build using the environment specification in llnl/tioga/spack.yaml; gcc should also work. Other compilers besides cce 16.0.1 untested. Note that you must run with the environment variable to enable gpu-aware cray-mpich, i.e. `export MPICH_GPU_SUPPORT_ENABLED=1`. The provided flux script (beatnik.flux) does this. diff --git a/configs/unm/README.md b/configs/unm/README.md deleted file mode 100644 index 34b76b2..0000000 --- a/configs/unm/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# UNM Beatnik Configurations -This directory contains run scripts for the UNM hopper cluster (using the CUP-ECS partition with A100 GPU nodes) and the UNM wheeler cluster. In general, beatnik should be installed on these systems using spack as described in the documentation in the parent directory of this directory. - - 1. Hopper GPU systems - `hopper.slurm` - 2. Wheeler CPU systems - `wheeler.slurm` - -Please read the comments at the top of each file to understand their requirements and how to use the files on these systems (e.g. which system modules to load along with the spack to make them work). diff --git a/configs/unm/hopper.slurm b/configs/unm/hopper/hopper.slurm similarity index 100% rename from configs/unm/hopper.slurm rename to configs/unm/hopper/hopper.slurm diff --git a/configs/unm/hopper/spack.yaml b/configs/unm/hopper/spack.yaml new file mode 100644 index 0000000..c86667c --- /dev/null +++ b/configs/unm/hopper/spack.yaml @@ -0,0 +1,21 @@ +spack: + specs: + - matrix: + - - cuda@11.7 + - ucx +ib_hw_tm +gdrcopy + - openmpi@4.1.4 +cuda cuda_arch=80 fabrics=ucx + - heffte +fftw +cuda cuda_arch=80 + - kokkos +hwloc +cuda +cuda_lambda +wrapper cuda_arch=80 + - cabana @master +cajita +heffte +mpi +cuda +arborx cuda_arch=80 + - ['%gcc@11.2.0'] + - hdf5 +mpi + - silo@4.11 +mpi + - blt + - cmake + concretizer: + unify: true + view: true + + upstreams: + carc-spack: + install_tree: /opt/spack/opt/spack \ No newline at end of file diff --git a/configs/unm/wheeler.slurm b/configs/unm/wheeler/wheeler.slurm similarity index 100% rename from configs/unm/wheeler.slurm rename to configs/unm/wheeler/wheeler.slurm