From 1f3381343148db800ac7ff3d829be4f0888502d1 Mon Sep 17 00:00:00 2001 From: JStewart28 <80227058+JStewart28@users.noreply.github.com> Date: Wed, 25 Oct 2023 16:11:38 -0600 Subject: [PATCH 1/6] Added spack.yaml for hopper --- configs/unm/spack_hopper.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 configs/unm/spack_hopper.yaml diff --git a/configs/unm/spack_hopper.yaml b/configs/unm/spack_hopper.yaml new file mode 100644 index 0000000..c86667c --- /dev/null +++ b/configs/unm/spack_hopper.yaml @@ -0,0 +1,21 @@ +spack: + specs: + - matrix: + - - cuda@11.7 + - ucx +ib_hw_tm +gdrcopy + - openmpi@4.1.4 +cuda cuda_arch=80 fabrics=ucx + - heffte +fftw +cuda cuda_arch=80 + - kokkos +hwloc +cuda +cuda_lambda +wrapper cuda_arch=80 + - cabana @master +cajita +heffte +mpi +cuda +arborx cuda_arch=80 + - ['%gcc@11.2.0'] + - hdf5 +mpi + - silo@4.11 +mpi + - blt + - cmake + concretizer: + unify: true + view: true + + upstreams: + carc-spack: + install_tree: /opt/spack/opt/spack \ No newline at end of file From d87e4b01c7e21ed58b12c369219984eb6dc68e16 Mon Sep 17 00:00:00 2001 From: JStewart28 <80227058+JStewart28@users.noreply.github.com> Date: Wed, 25 Oct 2023 16:15:24 -0600 Subject: [PATCH 2/6] Added spack.yaml for CARC systems, tested on Hopper --- configs/unm/spack.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 configs/unm/spack.yaml diff --git a/configs/unm/spack.yaml b/configs/unm/spack.yaml new file mode 100644 index 0000000..c86667c --- /dev/null +++ b/configs/unm/spack.yaml @@ -0,0 +1,21 @@ +spack: + specs: + - matrix: + - - cuda@11.7 + - ucx +ib_hw_tm +gdrcopy + - openmpi@4.1.4 +cuda cuda_arch=80 fabrics=ucx + - heffte +fftw +cuda cuda_arch=80 + - kokkos +hwloc +cuda +cuda_lambda +wrapper cuda_arch=80 + - cabana @master +cajita +heffte +mpi +cuda +arborx cuda_arch=80 + - ['%gcc@11.2.0'] + - hdf5 +mpi + - silo@4.11 +mpi + - blt + - cmake + concretizer: + unify: true + view: true + + upstreams: + carc-spack: + install_tree: /opt/spack/opt/spack \ No newline at end of file From 1731d05bd9494f04d0efe9184fbcf5e317e60b21 Mon Sep 17 00:00:00 2001 From: JStewart28 <80227058+JStewart28@users.noreply.github.com> Date: Wed, 25 Oct 2023 16:16:32 -0600 Subject: [PATCH 3/6] Delete configs/unm/spack_hopper.yaml Uploaded new spack.yaml without hopper in the name --- configs/unm/spack_hopper.yaml | 21 --------------------- 1 file changed, 21 deletions(-) delete mode 100644 configs/unm/spack_hopper.yaml diff --git a/configs/unm/spack_hopper.yaml b/configs/unm/spack_hopper.yaml deleted file mode 100644 index c86667c..0000000 --- a/configs/unm/spack_hopper.yaml +++ /dev/null @@ -1,21 +0,0 @@ -spack: - specs: - - matrix: - - - cuda@11.7 - - ucx +ib_hw_tm +gdrcopy - - openmpi@4.1.4 +cuda cuda_arch=80 fabrics=ucx - - heffte +fftw +cuda cuda_arch=80 - - kokkos +hwloc +cuda +cuda_lambda +wrapper cuda_arch=80 - - cabana @master +cajita +heffte +mpi +cuda +arborx cuda_arch=80 - - ['%gcc@11.2.0'] - - hdf5 +mpi - - silo@4.11 +mpi - - blt - - cmake - concretizer: - unify: true - view: true - - upstreams: - carc-spack: - install_tree: /opt/spack/opt/spack \ No newline at end of file From f5ef7d4b6e0b430956aa9bbdb10b698b9050d85e Mon Sep 17 00:00:00 2001 From: JStewart28 <80227058+JStewart28@users.noreply.github.com> Date: Wed, 25 Oct 2023 16:19:14 -0600 Subject: [PATCH 4/6] Added information about spack.yaml for CARC --- configs/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/configs/README.md b/configs/README.md index 201fa0a..ca53ab6 100644 --- a/configs/README.md +++ b/configs/README.md @@ -19,7 +19,8 @@ The beatnik spack package should enforce its build requirements appropriately; w We have tested beatnik installation on the following systems via either spack install with the provided spack install flags or a spack environment. An example run script is also provided for each of these systems in the appropriate subdirectory. * University of New Mexico - These systems simply use `spack install` as the UNM machines have a full global spack packages.yaml already set up. Because thse systems use spack for package maintenance, we recommend adding the provided `upstreams.yaml` file in the `unm` directory to the `.spack` directory in your home directory. * Hopper V100/A100 GPU cluster system - `spack install beatnik +cuda cuda_arch=80 %gcc ^cuda@11` (or `cuda_arch=70` for the V100 nodes); ^cuda@11 is needed to avoid using CUDA 12 which the UNM cuda drivers aren't updated to support as of 10/1/23. Only tested with gcc as the compiler. - * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is sufficient + * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is sufficient. + * If you prefer to use a spack environment to run beatnik instead of installing beatnik via spack, a `spack.yaml` file is provided in the `unm` directory. * Lawrence Livermore National Laboratory - These systems need a spack environment (provided) to set up compilers and external packages to use spack effectively on these systems. Use `spack env create` with the provided spack.yaml to build beatnik in an environment on these systems. Simple test run scripts are also provided. * Lassen V100 GPU system - Build using the environment specification in llnl/lassen/spack.yaml. Other compilers besides gcc untested. Note that cuda-aware MPI support is broken in spectrum MPI with cuda versions later than 11.2; the spack configuration avoids this, but if you're building beatnik by hand on Lassen or another IBM CORAL system, be aware of this limitation. * Tioga MX250X GPU system - Build using the environment specification in llnl/tioga/spack.yaml; gcc should also work. Other compilers besides cce 16.0.1 untested. Note that you must run with the environment variable to enable gpu-aware cray-mpich, i.e. `export MPICH_GPU_SUPPORT_ENABLED=1`. The provided flux script (beatnik.flux) does this. From 399f78f69bf0758b0b6e29ba85a664e225ac290f Mon Sep 17 00:00:00 2001 From: Patrick Bridges Date: Thu, 26 Oct 2023 10:59:48 -0600 Subject: [PATCH 5/6] Minor documentation/directory fixes for UNM configs --- configs/README.md | 7 +++---- configs/unm/README.md | 7 ------- configs/unm/{ => hopper}/hopper.slurm | 0 configs/unm/{ => hopper}/spack.yaml | 0 configs/unm/{ => wheeler}/wheeler.slurm | 0 5 files changed, 3 insertions(+), 11 deletions(-) delete mode 100644 configs/unm/README.md rename configs/unm/{ => hopper}/hopper.slurm (100%) rename configs/unm/{ => hopper}/spack.yaml (100%) rename configs/unm/{ => wheeler}/wheeler.slurm (100%) diff --git a/configs/README.md b/configs/README.md index ca53ab6..7985956 100644 --- a/configs/README.md +++ b/configs/README.md @@ -17,10 +17,9 @@ The beatnik spack package should enforce its build requirements appropriately; w ### Current list of tested systems targets and suggested installation method We have tested beatnik installation on the following systems via either spack install with the provided spack install flags or a spack environment. An example run script is also provided for each of these systems in the appropriate subdirectory. - * University of New Mexico - These systems simply use `spack install` as the UNM machines have a full global spack packages.yaml already set up. Because thse systems use spack for package maintenance, we recommend adding the provided `upstreams.yaml` file in the `unm` directory to the `.spack` directory in your home directory. - * Hopper V100/A100 GPU cluster system - `spack install beatnik +cuda cuda_arch=80 %gcc ^cuda@11` (or `cuda_arch=70` for the V100 nodes); ^cuda@11 is needed to avoid using CUDA 12 which the UNM cuda drivers aren't updated to support as of 10/1/23. Only tested with gcc as the compiler. - * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is sufficient. - * If you prefer to use a spack environment to run beatnik instead of installing beatnik via spack, a `spack.yaml` file is provided in the `unm` directory. + * University of New Mexico - UNM systems have a full global spack packages.yaml already set up, and so it is often possible to use `spack install` with the appropriate compiler and other flags to install beatnik on these systems. We recommend adding the provided `upstreams.yaml` file in the `unm` directory to the `.spack` directory in your home directory. + * Hopper V100/A100 GPU cluster system - Build using the environment specification in `unm/hopper/spack.yaml`. It is sometimes possible to simply use `spack install beatnik +cuda cuda_arch=80 %gcc ^cuda@11` (or `cuda_arch=70` for the V100 nodes), but this may not always work depending on how spack chooses to concretize the spack installation. + * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is generally sufficient. * Lawrence Livermore National Laboratory - These systems need a spack environment (provided) to set up compilers and external packages to use spack effectively on these systems. Use `spack env create` with the provided spack.yaml to build beatnik in an environment on these systems. Simple test run scripts are also provided. * Lassen V100 GPU system - Build using the environment specification in llnl/lassen/spack.yaml. Other compilers besides gcc untested. Note that cuda-aware MPI support is broken in spectrum MPI with cuda versions later than 11.2; the spack configuration avoids this, but if you're building beatnik by hand on Lassen or another IBM CORAL system, be aware of this limitation. * Tioga MX250X GPU system - Build using the environment specification in llnl/tioga/spack.yaml; gcc should also work. Other compilers besides cce 16.0.1 untested. Note that you must run with the environment variable to enable gpu-aware cray-mpich, i.e. `export MPICH_GPU_SUPPORT_ENABLED=1`. The provided flux script (beatnik.flux) does this. diff --git a/configs/unm/README.md b/configs/unm/README.md deleted file mode 100644 index 34b76b2..0000000 --- a/configs/unm/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# UNM Beatnik Configurations -This directory contains run scripts for the UNM hopper cluster (using the CUP-ECS partition with A100 GPU nodes) and the UNM wheeler cluster. In general, beatnik should be installed on these systems using spack as described in the documentation in the parent directory of this directory. - - 1. Hopper GPU systems - `hopper.slurm` - 2. Wheeler CPU systems - `wheeler.slurm` - -Please read the comments at the top of each file to understand their requirements and how to use the files on these systems (e.g. which system modules to load along with the spack to make them work). diff --git a/configs/unm/hopper.slurm b/configs/unm/hopper/hopper.slurm similarity index 100% rename from configs/unm/hopper.slurm rename to configs/unm/hopper/hopper.slurm diff --git a/configs/unm/spack.yaml b/configs/unm/hopper/spack.yaml similarity index 100% rename from configs/unm/spack.yaml rename to configs/unm/hopper/spack.yaml diff --git a/configs/unm/wheeler.slurm b/configs/unm/wheeler/wheeler.slurm similarity index 100% rename from configs/unm/wheeler.slurm rename to configs/unm/wheeler/wheeler.slurm From 0aa4599bd7e0d88beaa2197f9fa502930f4980ac Mon Sep 17 00:00:00 2001 From: JStewart28 <80227058+JStewart28@users.noreply.github.com> Date: Thu, 26 Oct 2023 11:04:03 -0600 Subject: [PATCH 6/6] Added 'the' on line 19 --- configs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configs/README.md b/configs/README.md index 7985956..727b729 100644 --- a/configs/README.md +++ b/configs/README.md @@ -16,7 +16,7 @@ The beatnik spack package should enforce its build requirements appropriately; w ### Current list of tested systems targets and suggested installation method -We have tested beatnik installation on the following systems via either spack install with the provided spack install flags or a spack environment. An example run script is also provided for each of these systems in the appropriate subdirectory. +We have tested the beatnik installation on the following systems via either spack install with the provided spack install flags or a spack environment. An example run script is also provided for each of these systems in the appropriate subdirectory. * University of New Mexico - UNM systems have a full global spack packages.yaml already set up, and so it is often possible to use `spack install` with the appropriate compiler and other flags to install beatnik on these systems. We recommend adding the provided `upstreams.yaml` file in the `unm` directory to the `.spack` directory in your home directory. * Hopper V100/A100 GPU cluster system - Build using the environment specification in `unm/hopper/spack.yaml`. It is sometimes possible to simply use `spack install beatnik +cuda cuda_arch=80 %gcc ^cuda@11` (or `cuda_arch=70` for the V100 nodes), but this may not always work depending on how spack chooses to concretize the spack installation. * General UNM (Wheeler/Hopper) CPU systems - `spack install beatnik` is generally sufficient.