managing-custom-jupyter-envs-with-singularity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Managing custom Jupyter environments with Singularity\n",
    "---\n",
    "In this tutorial we will cover the basic workflow for managing custom software environments for Jupyter Notebooks using Singularity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Singularity is a developing platform, so version matters **a lot**. The version we will be using is the latest development HEAD of 2.3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.3-HEAD.gadf5259\n"
     ]
    }
   ],
   "source": [
    "singularity --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pull the base image from ~~SingularityHub~~ DockerHub\n",
    "We maintain a base image on SingularityHub for running Jupyter* (https://singularity-hub.org/collections/440/). This image contains the minimum dependencies and configuration needed to run containerized Notebooks (standalone or JupyterHub-spawned), and is intended to serve as a base for user-built software environments.\n",
    "\n",
    "At this time though bootstrapping from SingularityHub is still an upcoming feature (https://github.com/singularityware/singularity/issues/833) so we will instead be using the `jupyter/base-notebook` docker container as a base image. The base-notebook is provided by the Jupyter Docker Stacks project (https://github.com/jupyter/docker-stacks), which provides pre-built stacks ready to be run standalone or behind JupyterHub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A basic pull\n",
    "Start by pulling the Jupyter base image from SingularityHub. We specifically want commit _ae885c0a6226_, so we'll specify that by adding the `:ae885c0a6226` tag to the end of the repo path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing Singularity image subsystem\n",
      "\u001b[0mOpening image file: jupyter-base.img\n",
      "\u001b[0mCreating 892MiB image\n",
      "\u001b[0mBinding image to loop\n",
      "\u001b[0mCreating file system within image\n",
      "\u001b[0mImage is done: jupyter-base.img\n",
      "\u001b[0mDocker image path: index.docker.io/jupyter/base-notebook:ae885c0a6226\n",
      "Cache folder set to /home/vagrant/.singularity/docker\n",
      "Importing: base Singularity environment\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:e0a742c2abfd5e2a6f8ed15b1c78e873cf9559b96a04204daf6de5df01e3124c.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:486cb8339a27635fa93dc47aa0c689326a0a7cce388966d16daf8d265436cf7f.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:dc6f0d824617ad8a5d1163a5b2084814665dd83156317ad06ccf14deb517a053.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:4f7a5649a30e3f318ce5d7e4dbcbbeb6c0938c4cbae4d4a641fe910562ff4978.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:672363445ad2c734e29221a6b47f4e614b5adc8a3cdca3364f62db2ed2bdff0c.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:b337aaee648d9f87e96fae8b24ae2dd887a2ded309b38dbee691fcdb040878cc.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:7f77b4eaa7ff7da43552cca1a34f9d16a1bbe90ba779d2c4355a4c4d1ab6dd0f.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:94e7bffb8046310f285abb93104a0cddd073c5a44ec7cfab05a3db6128fb3006.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:fa94652128009709e53b85cf57087d739108eaa98fd6e599343a32267deda620.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:79c35484f704c0792d062e5749c5df82fa85820bf072531daccefb532d88f294.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:f1756f88332d61a74c658fae7b6a5b8509e59b24e7f4ae7412bb183e23c17bc8.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:e73fef5319c18fa96bf5654b990f993622087a8caa56c2a5673db488a549ab1f.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:ea6e64c66b3a6af070c29a97433acaf94f875d99b4d43509f874686afa66a7d8.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:f3e50561218a351fb914efee8fb3bcb05eb619f70615520375e864b40a0227c4.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:f2de9a241e20a4ca75822db878a3eecf6afee5c4a96f099b76fe24306a51c1ca.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:6684df1dccb5994df682ca325871418c86770373ca27638919dff8dbb42b063a.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:1033de5a165ffa098da08b2da7325b577e5585d55bee04439033d0e72103bd64.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:ff59b2aa8f66f44d88ee7f218edd92bf0ced0d585b880ea6e9fc267100718f1b.tar.gz\n",
      "Importing: /home/vagrant/.singularity/metadata/sha256:583daeeee9e135a869cbe39637eecb8a3b3790096af37bfdf5a2d98b51016110.tar.gz\n",
      "Done. Container is at: jupyter-base.img\n"
     ]
    }
   ],
   "source": [
    "singularity pull --name \"jupyter-base.img\" docker://jupyter/base-notebook:ae885c0a6226"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There it is! Your container is good to go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]\n",
      "               [--paths] [--json]\n",
      "               [subcommand]\n",
      "\n",
      "Jupyter: Interactive Computing\n",
      "\n",
      "positional arguments:\n",
      "  subcommand     the subcommand to launch\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help     show this help message and exit\n",
      "  --version      show the jupyter command's version and exit\n",
      "  --config-dir   show Jupyter config dir\n",
      "  --data-dir     show Jupyter data dir\n",
      "  --runtime-dir  show Jupyter runtime dir\n",
      "  --paths        show all Jupyter paths. Add --json for machine-readable\n",
      "                 format.\n",
      "  --json         output paths as machine-readable json\n",
      "\n",
      "Available subcommands: bundlerextension kernelspec lab labextension labhub\n",
      "migrate nbconvert nbextension notebook run serverextension troubleshoot trust\n"
     ]
    }
   ],
   "source": [
    "singularity exec -e jupyter-base.img jupyter -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customizing the base image\n",
    "The base image is meant to capture the _minimum_ config and dependencies to run Jupyter Notebooks. Here we detail how to customize the base image to better suit your needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resize the image\n",
    "This image uses the default size set by Singularity _(image size + 200M of padding)_ which is great for quick builds and pulls, but it is likely you'll need more space to accommodate your custom software stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "669M -rwxr-xr-x.  1 vagrant vagrant 893M Sep 27 15:41 jupyter-base.img\n"
     ]
    }
   ],
   "source": [
    "ls -lsah | grep jupyter-base.img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When pulling from a Docker registry, you can use the `--size` flag to specify the built image size. Notice that Singularity isn't grabbing Docker layers from the registry, because the specified commit _(ae885c0a6226)_ has already been pulled. Singularity Docker cache is located in `$HOME/.singularity/docker`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing Singularity image subsystem\n",
      "\u001b[0mOpening image file: jupyter-ext.img\n",
      "\u001b[0mCreating 3000MiB image\n",
      "\u001b[0mBinding image to loop\n",
      "\u001b[0mCreating file system within image\n",
      "\u001b[0mImage is done: jupyter-ext.img\n",
      "\u001b[0mDocker image path: index.docker.io/jupyter/base-notebook:latest\n",
      "Cache folder set to /home/vagrant/.singularity/docker\n",
      "Importing: base Singularity environment\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:e0a742c2abfd5e2a6f8ed15b1c78e873cf9559b96a04204daf6de5df01e3124c.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:486cb8339a27635fa93dc47aa0c689326a0a7cce388966d16daf8d265436cf7f.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:dc6f0d824617ad8a5d1163a5b2084814665dd83156317ad06ccf14deb517a053.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:4f7a5649a30e3f318ce5d7e4dbcbbeb6c0938c4cbae4d4a641fe910562ff4978.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:672363445ad2c734e29221a6b47f4e614b5adc8a3cdca3364f62db2ed2bdff0c.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:b337aaee648d9f87e96fae8b24ae2dd887a2ded309b38dbee691fcdb040878cc.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:7f77b4eaa7ff7da43552cca1a34f9d16a1bbe90ba779d2c4355a4c4d1ab6dd0f.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:94e7bffb8046310f285abb93104a0cddd073c5a44ec7cfab05a3db6128fb3006.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:fa94652128009709e53b85cf57087d739108eaa98fd6e599343a32267deda620.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:79c35484f704c0792d062e5749c5df82fa85820bf072531daccefb532d88f294.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:f1756f88332d61a74c658fae7b6a5b8509e59b24e7f4ae7412bb183e23c17bc8.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:e73fef5319c18fa96bf5654b990f993622087a8caa56c2a5673db488a549ab1f.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:ea6e64c66b3a6af070c29a97433acaf94f875d99b4d43509f874686afa66a7d8.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:f3e50561218a351fb914efee8fb3bcb05eb619f70615520375e864b40a0227c4.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:f2de9a241e20a4ca75822db878a3eecf6afee5c4a96f099b76fe24306a51c1ca.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:6684df1dccb5994df682ca325871418c86770373ca27638919dff8dbb42b063a.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:1033de5a165ffa098da08b2da7325b577e5585d55bee04439033d0e72103bd64.tar.gz\n",
      "Importing: /home/vagrant/.singularity/docker/sha256:ff59b2aa8f66f44d88ee7f218edd92bf0ced0d585b880ea6e9fc267100718f1b.tar.gz\n",
      "Importing: /home/vagrant/.singularity/metadata/sha256:583daeeee9e135a869cbe39637eecb8a3b3790096af37bfdf5a2d98b51016110.tar.gz\n",
      "Done. Container is at: jupyter-ext.img\n"
     ]
    }
   ],
   "source": [
    "singularity pull --size 3000 --name \"jupyter-ext.img\" docker://jupyter/base-notebook:ae885c0a6226"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]\n",
      "               [--paths] [--json]\n",
      "               [subcommand]\n",
      "\n",
      "Jupyter: Interactive Computing\n",
      "\n",
      "positional arguments:\n",
      "  subcommand     the subcommand to launch\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help     show this help message and exit\n",
      "  --version      show the jupyter command's version and exit\n",
      "  --config-dir   show Jupyter config dir\n",
      "  --data-dir     show Jupyter data dir\n",
      "  --runtime-dir  show Jupyter runtime dir\n",
      "  --paths        show all Jupyter paths. Add --json for machine-readable\n",
      "                 format.\n",
      "  --json         output paths as machine-readable json\n",
      "\n",
      "Available subcommands: bundlerextension kernelspec lab labextension labhub\n",
      "migrate nbconvert nbextension notebook run serverextension troubleshoot trust\n"
     ]
    }
   ],
   "source": [
    "singularity exec -e jupyter-ext.img jupyter -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing software _(the quick way)_\n",
    "By default Singularity containers mounted as read-only volumes, which means you won't be able to add content or install software _(even as a privileged user)_, save for default or system-mounted paths. In order to add content you must run your Singularity command with the `--writable` flag.\n",
    "\n",
    "For an interactive shell into your container, use the `shell` subcommand. The command below also passes the `-e` flag, which tells Singularity to strip the host environment before entering the container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Singularity: Invoking an interactive shell within container...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sudo singularity shell -e --writable jupyter-ext.img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can use the `exec` subcommand to execute commands in your container without leaving your host environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching package metadata ...........\n",
      "Solving package specifications: .\n",
      "\n",
      "Package plan for installation in environment /opt/conda:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    cycler:           0.10.0-py36_0 conda-forge\n",
      "    dbus:             1.10.22-0     conda-forge\n",
      "    expat:            2.2.1-0       conda-forge\n",
      "    fontconfig:       2.12.1-4      conda-forge\n",
      "    freetype:         2.7-1         conda-forge\n",
      "    gettext:          0.19.7-1      conda-forge\n",
      "    glib:             2.51.4-0      conda-forge\n",
      "    gst-plugins-base: 1.8.0-0       conda-forge\n",
      "    gstreamer:        1.8.0-2       conda-forge\n",
      "    icu:              58.1-1        conda-forge\n",
      "    jpeg:             9b-1          conda-forge\n",
      "    libiconv:         1.14-4        conda-forge\n",
      "    libpng:           1.6.28-0      conda-forge\n",
      "    libxcb:           1.12-1        conda-forge\n",
      "    libxml2:          2.9.5-0       conda-forge\n",
      "    matplotlib:       2.0.2-py36_2  conda-forge\n",
      "    mkl:              2017.0.3-0    defaults   \n",
      "    numpy:            1.13.1-py36_0 defaults   \n",
      "    pcre:             8.39-0        conda-forge\n",
      "    pyqt:             5.6.0-py36_4  conda-forge\n",
      "    pytz:             2017.2-py36_0 conda-forge\n",
      "    qt:               5.6.2-3       conda-forge\n",
      "    sip:              4.18-py36_1   conda-forge\n",
      "    xorg-libxau:      1.0.8-3       conda-forge\n",
      "    xorg-libxdmcp:    1.1.2-3       conda-forge\n",
      "\n",
      "expat-2.2.1-0. 100% |################################| Time: 0:00:00   1.48 MB/s\n",
      "gettext-0.19.7 100% |################################| Time: 0:00:00   9.27 MB/s\n",
      "icu-58.1-1.tar 100% |################################| Time: 0:00:01  21.29 MB/s\n",
      "jpeg-9b-1.tar. 100% |################################| Time: 0:00:00  25.47 MB/s\n",
      "libiconv-1.14- 100% |################################| Time: 0:00:00  16.77 MB/s\n",
      "mkl-2017.0.3-0 100% |################################| Time: 0:00:03  35.19 MB/s\n",
      "pcre-8.39-0.ta 100% |################################| Time: 0:00:00   1.21 MB/s\n",
      "xorg-libxau-1. 100% |################################| Time: 0:00:00  22.14 MB/s\n",
      "xorg-libxdmcp- 100% |################################| Time: 0:00:00  20.33 MB/s\n",
      "dbus-1.10.22-0 100% |################################| Time: 0:00:00   3.40 MB/s\n",
      "glib-2.51.4-0. 100% |################################| Time: 0:00:00  13.82 MB/s\n",
      "libpng-1.6.28- 100% |################################| Time: 0:00:00  68.13 MB/s\n",
      "libxcb-1.12-1. 100% |################################| Time: 0:00:00  11.46 MB/s\n",
      "libxml2-2.9.5- 100% |################################| Time: 0:00:00  12.21 MB/s\n",
      "freetype-2.7-1 100% |################################| Time: 0:00:00  14.38 MB/s\n",
      "gstreamer-1.8. 100% |################################| Time: 0:00:00  21.42 MB/s\n",
      "fontconfig-2.1 100% |################################| Time: 0:00:00  15.21 MB/s\n",
      "gst-plugins-ba 100% |################################| Time: 0:00:00  24.25 MB/s\n",
      "numpy-1.13.1-p 100% |################################| Time: 0:00:00  36.10 MB/s\n",
      "pytz-2017.2-py 100% |################################| Time: 0:00:00  48.89 MB/s\n",
      "sip-4.18-py36_ 100% |################################| Time: 0:00:00   7.02 MB/s\n",
      "cycler-0.10.0- 100% |################################| Time: 0:00:00  22.60 MB/s\n",
      "qt-5.6.2-3.tar 100% |################################| Time: 0:00:02  20.00 MB/s\n",
      "pyqt-5.6.0-py3 100% |################################| Time: 0:00:00  10.44 MB/s\n",
      "matplotlib-2.0 100% |################################| Time: 0:00:00  21.28 MB/s\n",
      "Fetching package metadata ...........\n",
      "Solving package specifications: .\n",
      "\n",
      "Package plan for installation in environment /opt/conda:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    libgfortran: 3.0.0-1            defaults   \n",
      "    pandas:      0.20.3-py36_1      conda-forge\n",
      "    patsy:       0.4.1-py36_0       conda-forge\n",
      "    scipy:       0.19.1-np113py36_0 defaults   \n",
      "    seaborn:     0.8.1-py36_0       conda-forge\n",
      "    statsmodels: 0.8.0-py36_0       conda-forge\n",
      "\n",
      "libgfortran-3. 100% |################################| Time: 0:00:00  17.90 MB/s\n",
      "scipy-0.19.1-n 100% |################################| Time: 0:00:01  35.76 MB/s\n",
      "pandas-0.20.3- 100% |################################| Time: 0:00:01  16.39 MB/s\n",
      "patsy-0.4.1-py 100% |################################| Time: 0:00:00  58.21 MB/s\n",
      "statsmodels-0. 100% |################################| Time: 0:00:00  11.16 MB/s\n",
      "seaborn-0.8.1- 100% |################################| Time: 0:00:00  55.75 MB/s\n"
     ]
    }
   ],
   "source": [
    "singularity exec -e --writable jupyter-ext.img /opt/conda/bin/conda install -y matplotlib\n",
    "singularity exec -e --writable jupyter-ext.img /opt/conda/bin/conda install -y seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now seaborn is installed in your image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seaborn                   0.8.1                    py36_0    conda-forge\n"
     ]
    }
   ],
   "source": [
    "singularity exec -e jupyter-ext.img conda list | grep seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Software _(the reproducible way)_\n",
    "Shelling into your container and making ad-hoc changes is excellent for debugging and initial development, but it is considered bad practice as the steps needed to construct your software environment are not captured and cannot be reproduced.\n",
    "\n",
    "To make durable, reproducible changes you need to build a spec file from which you can bootstrap your container. Bootstrapping must be done by a privileged user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BootStrap: docker\n",
      "From: jupyter/base-notebook\n",
      "\n",
      "%environment\n",
      "  export PATH=/opt/conda/bin:$PATH\n",
      "\n",
      "%post\n",
      "  export PATH=/opt/conda/bin:$PATH\n",
      "  echo \"Installing seaborn...\"\n",
      "  conda install matplotlib\n",
      "  conda install seaborn\n"
     ]
    }
   ],
   "source": [
    "cat jupyter-bootstrapped.def"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing Singularity image subsystem\n",
      "\u001b[0mOpening image file: jupyter-bootstrapped.img\n",
      "\u001b[0mCreating 2500MiB image\n",
      "\u001b[0mBinding image to loop\n",
      "\u001b[0mCreating file system within image\n",
      "\u001b[0mImage is done: jupyter-bootstrapped.img\n",
      "\u001b[0mSanitizing environment\n",
      "\u001b[0mBuilding from bootstrap definition recipe\n",
      "\u001b[0mAdding base Singularity environment to container\n",
      "Docker image path: index.docker.io/jupyter/base-notebook:latest\n",
      "Cache folder set to /root/.singularity/docker\n",
      "Exploding layer: sha256:e0a742c2abfd5e2a6f8ed15b1c78e873cf9559b96a04204daf6de5df01e3124c.tar.gz\n",
      "Exploding layer: sha256:486cb8339a27635fa93dc47aa0c689326a0a7cce388966d16daf8d265436cf7f.tar.gz\n",
      "Exploding layer: sha256:dc6f0d824617ad8a5d1163a5b2084814665dd83156317ad06ccf14deb517a053.tar.gz\n",
      "Exploding layer: sha256:4f7a5649a30e3f318ce5d7e4dbcbbeb6c0938c4cbae4d4a641fe910562ff4978.tar.gz\n",
      "Exploding layer: sha256:672363445ad2c734e29221a6b47f4e614b5adc8a3cdca3364f62db2ed2bdff0c.tar.gz\n",
      "Exploding layer: sha256:b337aaee648d9f87e96fae8b24ae2dd887a2ded309b38dbee691fcdb040878cc.tar.gz\n",
      "Exploding layer: sha256:7f77b4eaa7ff7da43552cca1a34f9d16a1bbe90ba779d2c4355a4c4d1ab6dd0f.tar.gz\n",
      "Exploding layer: sha256:94e7bffb8046310f285abb93104a0cddd073c5a44ec7cfab05a3db6128fb3006.tar.gz\n",
      "Exploding layer: sha256:fa94652128009709e53b85cf57087d739108eaa98fd6e599343a32267deda620.tar.gz\n",
      "Exploding layer: sha256:79c35484f704c0792d062e5749c5df82fa85820bf072531daccefb532d88f294.tar.gz\n",
      "Exploding layer: sha256:f1756f88332d61a74c658fae7b6a5b8509e59b24e7f4ae7412bb183e23c17bc8.tar.gz\n",
      "Exploding layer: sha256:e73fef5319c18fa96bf5654b990f993622087a8caa56c2a5673db488a549ab1f.tar.gz\n",
      "Exploding layer: sha256:ea6e64c66b3a6af070c29a97433acaf94f875d99b4d43509f874686afa66a7d8.tar.gz\n",
      "Exploding layer: sha256:f3e50561218a351fb914efee8fb3bcb05eb619f70615520375e864b40a0227c4.tar.gz\n",
      "Exploding layer: sha256:f2de9a241e20a4ca75822db878a3eecf6afee5c4a96f099b76fe24306a51c1ca.tar.gz\n",
      "Exploding layer: sha256:6684df1dccb5994df682ca325871418c86770373ca27638919dff8dbb42b063a.tar.gz\n",
      "Exploding layer: sha256:1033de5a165ffa098da08b2da7325b577e5585d55bee04439033d0e72103bd64.tar.gz\n",
      "Exploding layer: sha256:ff59b2aa8f66f44d88ee7f218edd92bf0ced0d585b880ea6e9fc267100718f1b.tar.gz\n",
      "Exploding layer: sha256:0ec5668b017f4b07564d4b53a6f4bd19c61dc4892342ed092c076c99ad065c2e.tar.gz\n",
      "Running post scriptlet\n",
      "+ export PATH=/opt/conda/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin\n",
      "+ echo Installing seaborn...\n",
      "Installing seaborn...\n",
      "+ conda install matplotlib\n",
      "Fetching package metadata ...........\n",
      "Solving package specifications: .\n",
      "\n",
      "Package plan for installation in environment /opt/conda:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    cycler:           0.10.0-py36_0 conda-forge\n",
      "    dbus:             1.10.22-0     conda-forge\n",
      "    expat:            2.2.1-0       conda-forge\n",
      "    fontconfig:       2.12.1-4      conda-forge\n",
      "    freetype:         2.7-1         conda-forge\n",
      "    gettext:          0.19.7-1      conda-forge\n",
      "    glib:             2.51.4-0      conda-forge\n",
      "    gst-plugins-base: 1.8.0-0       conda-forge\n",
      "    gstreamer:        1.8.0-2       conda-forge\n",
      "    icu:              58.1-1        conda-forge\n",
      "    jpeg:             9b-1          conda-forge\n",
      "    libiconv:         1.14-4        conda-forge\n",
      "    libpng:           1.6.28-0      conda-forge\n",
      "    libxcb:           1.12-1        conda-forge\n",
      "    libxml2:          2.9.5-0       conda-forge\n",
      "    matplotlib:       2.0.2-py36_2  conda-forge\n",
      "    mkl:              2017.0.3-0    defaults   \n",
      "    numpy:            1.13.1-py36_0 defaults   \n",
      "    pcre:             8.39-0        conda-forge\n",
      "    pyqt:             5.6.0-py36_4  conda-forge\n",
      "    pytz:             2017.2-py36_0 conda-forge\n",
      "    qt:               5.6.2-3       conda-forge\n",
      "    sip:              4.18-py36_1   conda-forge\n",
      "    xorg-libxau:      1.0.8-3       conda-forge\n",
      "    xorg-libxdmcp:    1.1.2-3       conda-forge\n",
      "\n",
      "Proceed ([y]/n)? \n",
      "expat-2.2.1-0. 100% |################################| Time: 0:00:00   1.50 MB/s\n",
      "gettext-0.19.7 100% |################################| Time: 0:00:00   7.85 MB/s\n",
      "icu-58.1-1.tar 100% |################################| Time: 0:00:01  23.17 MB/s\n",
      "jpeg-9b-1.tar. 100% |################################| Time: 0:00:00  20.14 MB/s\n",
      "libiconv-1.14- 100% |################################| Time: 0:00:00  12.02 MB/s\n",
      "mkl-2017.0.3-0 100% |################################| Time: 0:00:03  35.69 MB/s\n",
      "pcre-8.39-0.ta 100% |################################| Time: 0:00:00   1.31 MB/s\n",
      "xorg-libxau-1. 100% |################################| Time: 0:00:00  18.72 MB/s\n",
      "xorg-libxdmcp- 100% |################################| Time: 0:00:00  23.09 MB/s\n",
      "dbus-1.10.22-0 100% |################################| Time: 0:00:00   2.58 MB/s\n",
      "glib-2.51.4-0. 100% |################################| Time: 0:00:00   9.27 MB/s\n",
      "libpng-1.6.28- 100% |################################| Time: 0:00:00  55.61 MB/s\n",
      "libxcb-1.12-1. 100% |################################| Time: 0:00:00  11.54 MB/s\n",
      "libxml2-2.9.5- 100% |################################| Time: 0:00:00  17.10 MB/s\n",
      "freetype-2.7-1 100% |################################| Time: 0:00:00  23.24 MB/s\n",
      "gstreamer-1.8. 100% |################################| Time: 0:00:00   9.55 MB/s\n",
      "fontconfig-2.1 100% |################################| Time: 0:00:00   5.83 MB/s\n",
      "gst-plugins-ba 100% |################################| Time: 0:00:00   8.40 MB/s\n",
      "numpy-1.13.1-p 100% |################################| Time: 0:00:00  41.53 MB/s\n",
      "pytz-2017.2-py 100% |################################| Time: 0:00:00   4.68 MB/s\n",
      "sip-4.18-py36_ 100% |################################| Time: 0:00:00   6.29 MB/s\n",
      "cycler-0.10.0- 100% |################################| Time: 0:00:00  23.92 MB/s\n",
      "qt-5.6.2-3.tar 100% |################################| Time: 0:00:02  21.20 MB/s\n",
      "pyqt-5.6.0-py3 100% |################################| Time: 0:00:00  19.36 MB/s\n",
      "matplotlib-2.0 100% |################################| Time: 0:00:00  20.04 MB/s\n",
      "+ conda install seaborn\n",
      "Fetching package metadata ...........\n",
      "Solving package specifications: .\n",
      "\n",
      "Package plan for installation in environment /opt/conda:\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "    libgfortran: 3.0.0-1            defaults   \n",
      "    pandas:      0.20.3-py36_1      conda-forge\n",
      "    patsy:       0.4.1-py36_0       conda-forge\n",
      "    scipy:       0.19.1-np113py36_0 defaults   \n",
      "    seaborn:     0.8.1-py36_0       conda-forge\n",
      "    statsmodels: 0.8.0-py36_0       conda-forge\n",
      "\n",
      "Proceed ([y]/n)? \n",
      "libgfortran-3. 100% |################################| Time: 0:00:00   4.70 MB/s\n",
      "scipy-0.19.1-n 100% |################################| Time: 0:00:01  29.71 MB/s\n",
      "pandas-0.20.3- 100% |################################| Time: 0:00:01  12.90 MB/s\n",
      "patsy-0.4.1-py 100% |################################| Time: 0:00:00  10.76 MB/s\n",
      "statsmodels-0. 100% |################################| Time: 0:00:00  14.35 MB/s\n",
      "seaborn-0.8.1- 100% |################################| Time: 0:00:00  55.54 MB/s\n",
      "Adding environment to container\n",
      "Finalizing Singularity container\n"
     ]
    }
   ],
   "source": [
    "singularity create --force --size 2500 jupyter-bootstrapped.img\n",
    "sudo /usr/local/bin/singularity bootstrap jupyter-bootstrapped.img jupyter-bootstrapped.def"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seaborn                   0.8.1                    py36_0    conda-forge\n"
     ]
    }
   ],
   "source": [
    "singularity exec -e jupyter-bootstrapped.img conda list | grep seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using your environment in a notebook\n",
    "This next section will cover basic strategies for using your very new, very custom software environment in a Jupyter Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom kernels\n",
    "IPython notebooks interface with the system via an abstraction called _Kernels_. A wide variety of languages are supported via Kernels, and they can be customized by editing the kernelspec JSON file that defines them. Here is the default Python 3 kernelspec for reference:\n",
    "```json\n",
    "\"argv\": [\n",
    "  \"python\",\n",
    "  \"-m\",\n",
    "  \"ipykernel_launcher\",\n",
    "  \"-f\",\n",
    "  \"{connection_file}\"\n",
    " ],\n",
    " \"display_name\": \"Python 3\",\n",
    " \"language\": \"python\"\n",
    "}\n",
    "```\n",
    "\n",
    "The `argv` key in this JSON object is the list that Jupyter uses to construct the kernel command when a notebook is started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the `singularity exec` subcommand? We can leverage that here to start a kernel in our container from a notebook server running in our host environment. All we need to do is prepend the components of the exec command to the `argv` list:\n",
    "```json\n",
    "\"argv\": [\n",
    "  \"singularity\",\n",
    "  \"exec\",\n",
    "  \"-e\",\n",
    "  \"jupyter-bootstrapped.img\",\n",
    "  \"python\",\n",
    "  \"-m\",\n",
    "  \"ipykernel_launcher\",\n",
    "  \"-f\",\n",
    "  \"{connection_file}\"\n",
    " ],\n",
    " \"display_name\": \"Python 3\",\n",
    " \"language\": \"python\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating a new kernel\n",
    "We'll start by generating a new kernelspec in a temporary location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[InstallIPythonKernelSpecApp] WARNING | Installing to /tmp/share/jupyter/kernels, which is not in ['/home/vagrant/.local/share/jupyter/kernels', '/home/vagrant/venv-jupyter/share/jupyter/kernels', '/usr/local/share/jupyter/kernels', '/usr/share/jupyter/kernels', '/home/vagrant/.ipython/kernels']. The kernelspec may not be found.\n",
      "Installed kernelspec python3 in /tmp/share/jupyter/kernels/python3\n"
     ]
    }
   ],
   "source": [
    "ipython kernel install --prefix /tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Editing the kernel\n",
    "Now edit your kernelspec. An example can be found in this repo at <a href=\"/user/vagrant/edit/singularity-jupyter-demo/singularity-kernel.json\">singularity-kernel.json</a>. Make sure to rename the kernelspec directory to avoid conflicts with existing kernels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mv /tmp/share/jupyter/kernels/python3 /tmp/share/jupyter/kernels/seaborn\n",
    "# Then edit /tmp/share/jupyter/kernels/seaborn/kernel.json (in our case we'll just copy the example)\n",
    "cp singularity-kernel.json /tmp/share/jupyter/kernels/seaborn/kernel.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Install the kernel\n",
    "Finish by installing your new kernel to a location where your notebook will look when it starts. The `--user` flag specifies that you wish to install the kernel only for your user, and prevents the install from attempting to use `sys.prefix`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[InstallKernelSpec] Removing existing kernelspec in /home/vagrant/.local/share/jupyter/kernels/seaborn\n",
      "[InstallKernelSpec] Installed kernelspec seaborn in /home/vagrant/.local/share/jupyter/kernels/seaborn\n"
     ]
    }
   ],
   "source": [
    "jupyter kernelspec install --user /tmp/share/jupyter/kernels/seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IPython Parallel\n",
    "A similar approach can be used to execute ipengines on Summit, providing a mechanism for multi-node ipyparallel workflows that run in Singularity containers. This approach is detailed in the _ipyparallel-using-singularity_ notebook in this directory."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}