ipyparallel-using-singularity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IPython Parallel with Singularity\n",
    "---\n",
    "Singularity containers can be used to execute ipengines on Summit, providing a mechanism for multi-node ipyparallel workflows that run in custom software environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pull the base image\n",
    "Since bootstrapping from Singularity Hub is currently unsupported, we will instead pull the CURC base image from our Docker Hub organization _(let me know if you'd like to be added to it)_. This base image includes `ipyparallel==6.0.2`.\n",
    "\n",
    "Before we pull, we will also change the directory Singularity uses for caching, to avoid filling up our home directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml python/3.5.1 singularity/2.3.2\n",
    "export SINGULARITY_CACHEDIR=/projects/$USER/.singularity\n",
    "singularity pull --name \"jupyter-ipyparallel.img\" docker://researchcomputing/jupyter-base"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Make a new IPython parallel profile\n",
    "Now that the image has been pulled, we need to make a new IPython parallel profile to work with. If you wish to start from scratch, run\n",
    "```\n",
    "ipython profile create --parallel --profile=singularity-shas\n",
    "```\n",
    "\n",
    "It is quickest to start from the example profile we provide in your home directory though:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extensions    profile_default\t    profile_singularity-shas\n",
      "nbextensions  profile_example-shas\n"
     ]
    }
   ],
   "source": [
    "cp -a ~/.ipython/profile_example-shas ~/.ipython/profile_singularity-shas\n",
    "ls ~/.ipython/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Edit the cluster config\n",
    "We need to edit the cluster config _(located at `~/.ipython/profile_singularity-shas/ipcluster_config.py`)_ to use `singularity exec`.\n",
    "\n",
    "```python\n",
    "# Configuration file for ipcluster.\n",
    "\n",
    "c.IPClusterEngines.engine_launcher_class = 'ipyparallel.apps.launcher.SlurmEngineSetLauncher'\n",
    "c.SlurmLauncher.qos = 'normal'\n",
    "c.SlurmLauncher.timelimit = '4:00:00'\n",
    "# c.SlurmLauncher.account = ''\n",
    "# c.SlurmLauncher.machines = ''\n",
    "# c.SlurmLauncher.mem = ''\n",
    "# c.SlurmLauncher.resources = ''\n",
    "\n",
    "c.SlurmEngineSetLauncher.batch_template = \"\"\"#!/bin/bash\n",
    "\n",
    "#SBATCH --partition shas\n",
    "#SBATCH --qos {qos}\n",
    "#SBATCH --job-name singularity-ipengine\n",
    "#SBATCH --ntasks {n}\n",
    "# This will run a single ipengine per CPU\n",
    "#SBATCH --cpus-per-task 1\n",
    "# Use ntasks-per-node=1 to run one ipengine per node\n",
    "#SBATCH --time {timelimit}\n",
    "#SBATCH --output {profile_dir}/log/slurm.out\n",
    "\n",
    "module load singularity/2.3.2 impi\n",
    "\n",
    "# This line executes each ipengine in a Singularity container, and aggregates it into an MPI pool.\n",
    "mpirun singularity exec -e /projects/sampedro/singularity-jupyter-demo/jupyter-rc.img ipengine --profile-dir=\"{profile_dir}\" --cluster-id=\"{cluster_id}\"\n",
    "\"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}