-
Notifications
You must be signed in to change notification settings - Fork 6
Getting meraxes working on ozstar
These instructions were kindly written by James Davies (The University of Melbourne).
Make sure you have the following modules loaded:
- openmpi/3.0.0
- cmake/3.10.2
- icc/2016.2.181-gcc-6.4.0
- git/2.16.0
- gsl/2.4
- hdf5/1.10.1
- fftw/3.3.7
The module gcccore/6.4.0 will be automatically loaded by icc. Previously we had to unload it for meraxes to work, but now it must be left loaded. Also, load the modules manually in your bash profile, or use a custom script, as using module restore can cause some issues.
Clone your remote repo:
git clone github.com/username/meraxes.git #username for whichever fork you need
Checkout your working branch:
git checkout -b <working_branch_name>
Create the file local.cmake in src directory with the following text:
set(CMAKE_C_COMPILER "icc")
set(CMAKE_CXX_COMPILER "icpc")
set(CMAKE_CXX_FLAGS "-wd3802")
The first line lets the compiler know to use the correct hdf5, if you have anaconda loaded, for example. this can also be fixed by loading hdf5 after anaconda, as this will put hdf5 before anaconda in the paths.
The second line downloads a version of FFTW that works with meraxes, as ozstar does not have the right module yet.
Make a build directory inside meraxes/src/, cd in
Execute cmake ../ from the build directory, then ccmake ../ to configure your makefile (use CUDA or not, etc)
Type make to compile the code
If you just want to get started, typing make in src/ instead of the last 3 steps (just after creating local.cmake) will make a folder named "build" and compile meraxes there with default options.
If you get this error:
../3rd_party/lib/libfftw3f_mpi.so: error: undefined reference to '__intel_sse2_strlen'
../3rd_party/lib/libfftw3f_mpi.so: error: undefined reference to '_intel_fast_memcpy'
../3rd_party/lib/libfftw3f_mpi.so: error: undefined reference to '_intel_fast_memmove'
../3rd_party/lib/libfftw3f_mpi.so: error: undefined reference to '_intel_fast_memset'
collect2: error: ld returned 1 exit status
make[3]: *** [bin/meraxes] Error 1
make[2]: *** [CMakeFiles/meraxes.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [default] Error 2
Then cmake is probably finding the wrong compiler, check your modules for any extra compilers apart from the ones listed above, and unload them if necessary.
Note: when recompliling Meraxes against different libraries, remove your build directory and remake it before running cmake ../
Create an output directory in your project folder
Running requires two files: a snaplist with desired output snapshots all on one line (0-164 for Tiamat, using seq with the -s ' ' option to generate a list makes this easier), and a parameter file which overrides options in a specified defaults file (eg: Tiamat.par in /meraxes/inputs)
The easiest way to obtain these is to use the setuprun makefile target. e.g.:
RUNDIR=/path/where/you/are/going/to/run/meraxes make setuprunThis target will make the directory specified by RUNDIR and copy the necessary files there.
If, instead, you prefer to setup your run manually you should use the input.par file located in the build directory, and copy this file to the directory where you will be running the code. Note that you should not use the input.par file located in the input directory. This is a template file which is populated during the compilation phase.
You can also change any of the other parameters in the defaults file by copying the line into the input.par parameter file and altering it.
For very short testing runs on the interactive/login nodes, use mpirun as usual
mpirun -n <number_of_cores> <path_to_executable> <path_to_parameter_file>
Use sbatch to submit batch jobs to compute nodes
sbatch <script_name>
#!/bin/bash
#SBATCH --job-name=example
#SBATCH --nodes=1 #number of nodes requested, use 1 to keep everything on the same node
#SBATCH --ntasks=8 #number of cpus required
#SBATCH --time=1:00:00 #after this time the job will be killed
#SBATCH --mem-per-cpu=1000 #memory in MB per core
#SBATCH --partition=skylake #skylake is the default partition, should automatically detect and move jobs to gpu if needed
#SBATCH --gres=gpu:1 #gpu allocation (THIS LINE IS ONLY REQUIRED IF MERAXES IS BUILT WITH GPU SUPPORT)
#SBATCH -D workdir #working directory
#SBATCH -o outfile #output file
#SBATCH -e errfile #error file, will go to outfile if omitted
#load any modules you need here, I recommend making a shell script and calling it like this
source ~/.modules.sh
srun <path_to_meraxes_executable> <path_to_parameter_file>
Note: you can request a maximum of 5968MB of memory per cpu, or 191600MB in total for standard jobs
srun, mpirun and mpiexec apparently all work when executing MPI jobs in the queue, however the docs use srun so that's what I'll be using here.
The error and outfiles specified are updated in real time with results from the job, so they can be read and checked as the job progesses.
srun can also be used on the command line to submit jobs to compute nodes with realtime output to your login shell. For example, the following line requests 16 cpus, with 2GB of memory each for 4 hours:
srun --time=4:0:0 --nodes=1 --ntasks=4 --mem-per-cpu=2G ./my-interactive-mpi-program
The "sinteractive" command is used to request an interactive shell on a compute node, using the same flags as srun or a batch script, but without the last argument/executable
For more info on slurm and #SBATCH options, see the ozstar docs.
To keep us all up to date, and our forks tidy, we should be keeping our own master branches even with smutch/master, and working off another branch. This means that we should be updating our master branch every so often, and then rebasing our working branch to apply all the changes in order and keep our commit histories consistent.
Set up an upstream remote repo to smutch/meraxes
git remote add upstream https://github.com/smutch/meraxes.git
Merge changes into your local master branch
git checkout master
git pull --ff upstream master
Push changes to your fork (assuming your fork remote is named "origin")
git push origin master
Switch to your branch, then use git rebase
git checkout <working_branch>
git rebase master
Any conflicts between new commits to master and your working branch are displayed and must be manually dealt with before the rebase can be completed
Push the rebased branches to your fork (assuming your fork remote is named "origin")
git push -f origin <working_branch>
For more info on forking and updating, see this helpful document.