01 - Jupyter, Sandboxes, and Logging In.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jupyter environment\n",
    "\n",
    "Jupyter is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more.\n",
    "\n",
    "[Website](http://jupyter.org/) [Project repository](https://github.com/jupyter)\n",
    "\n",
    "We will be using a Jupyter server as the primary web interface for this workshop. Several notebooks have been provided to you, in advance, to guide you through the workshop. After the workshop, you may use the [Agave Jupyter image](https://hub.docker.com/r/agaveplatform/jupyter-notebook/) to recreate the notebook server and repeat the workshop, or continue on with your own work at your leisure.  \n",
    "\n",
    "The Agave image has several customizations to facilitate use of the platform and ease much of the heavy lifting done behind the scenes in this tutorial.\n",
    "\n",
    "\n",
    "### Custom Kernels\n",
    "\n",
    "Your Jupyter server has multiple kernels available for use right away. We have preconfigured them with several useful libraries and tools to help users get up and running with common tasks easier. Additionally, we have bundled in Agave CLI and Python & R SDK into the Bash, Python 2, Python 3, and R kernels respectively. Each kernel is pre-authenticated with valid Agave auth tokens that you can use to begin interacting with the Agave Platform right away.\n",
    "\n",
    "\n",
    "### Shared file system\n",
    "\n",
    "Your home directory on the Jupyter server is shared with your sandbox, so you can safely copy data between the two environments quickly and easily.\n",
    "\n",
    "\n",
    "### Web console  \n",
    "\n",
    "Jupyter contains a web terminal that can be used to access your sandbox environment or interact with the Jupyter container itself. To login to your sandbox from the Jupyter web terminal, simply run the following command:  \n",
    "\n",
    "```\n",
    "ssh -p 10022 $VM_IPADDRESS\n",
    "```  \n",
    "\n",
    "### Tutorial notebooks  \n",
    "\n",
    "This tutorial is presented as a series of Jupyter notebooks. If you are attending this tutorial in person, you will download the notebooks into the home directory of your notebook server. If you are following along after the fact, you should download the notebooks from the github repository into your Jupyter workspace.\n",
    "\n",
    "```\n",
    "git clone --depth 1 https://github.com/agavetraining/sc18\n",
    "``` \n",
    "\n",
    "### API access  \n",
    "\n",
    "The tutorial walks you through the process of obtaining a set of API keys an authenticating to the Agave Platform. Once this is done, you no longer need to authenticate to follow the tutorial. Both the Agave CLI and Python SDK will be picked up your authorization cache and automatically refresh it as needed.\n",
    "\n",
    "### Extras\n",
    "\n",
    "Inside of the `examples` directory, you will find several notebooks to help you learn more about the Agave platform, containers, and SciOps. We leave these for you to follow after the tutorial.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sandbox environment\n",
    "\n",
    "The tutorial sandbox is an Ubuntu 16.04 server running as a Docker container on a VM dedicated for your use in this tutorial. The sandbox has a standard HPC build environment with OpenMPI, Python 2, Python 3, build-essential, gfortran, openssl, git, jq, vim, and a host of other utilities. \n",
    "\n",
    "### Container runtimes  \n",
    "\n",
    "Docker and Singularity are both pre-installed in your Sandbox. All images used in this tutorial are available from the public Agave Docker Hub and Singularity Hub accounts. You may also use your own private registry accounts. You will need to login to the respective registries on your own.\n",
    "\n",
    "\n",
    "### Funwave example code  \n",
    "\n",
    "The sample code for this project is already present in `$HOME/FUNWAVE-TVD`.\n",
    "\n",
    "\n",
    "### Shared file system\n",
    "\n",
    "Your `$HOME/work` directory on the Jupyter server is shared with your sandbox, so you can safely copy data between the two environments quickly and easily.\n",
    "\n",
    "\n",
    "### Accessibility  \n",
    "\n",
    "To login to the sandbox from outside the Jupyter server, use the host IP address. You will find the public IP address of your sandbox in the `$VM_IPADDRESS` environment variable. Valid ssh keys are available in the `~/.ssh` director of your Jupyter server. Alternatively, you can append your own public key to the `$HOME/.ssh/authorized_keys` file.  \n",
    "\n",
    "```\n",
    "ssh -i /path/to/private/key.pem -p 10022 jovyan@$VM_IPADDRESS\n",
    "```  \n",
    "\n",
    "### Persistence\n",
    "\n",
    "Your VM will remain available for 1-2 days following the tutorial. During that time, your data will remain available.  After that, the VM an any data saved with it will be destroyed. If you need to persist your data, it is recommended that you move it to another host, or [create your own account](https://public.agaveapi.co/create_account) in the Agave public tenant and save your data in the free cloud storage provied to you by default there.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jenkins environment\n",
    "\n",
    "The tutorial CI server is a Jenkins LTS server running as a Docker container on a VM dedicated for your use in this tutorial. The server is preconfigured with several popular plugins such as pipeline, github, credentials, and pipeline steps, which are used during this tutorial.\n",
    "\n",
    "### Build environment\n",
    "\n",
    "The Jenkins host has the Agave CLI preinstalled with a valid set of client keys and auth token. This will allow you to create persistent Freestyle and Pipeline jobs that leverage the Agave CLI without worrying about authenticating or refreshing your token. \n",
    "\n",
    "The SSH keys created and installed into your jupyter and sandbox environments have also been installed as the default ssh credentials for the jenkins server. This will allow your jenkins server to access both your sandbox and jupyter hosts as needed.\n",
    "\n",
    "\n",
    "### Persistent home directory\n",
    "\n",
    "Your `JENKINS_HOME` directory is persisted as an external Docker volume to perserve data between restarts. There is no shared file system between the Jenkins container and the Jupyter or Sandbox containers.\n",
    "\n",
    "\n",
    "### Accessibility  \n",
    "\n",
    "Direct SSH into the Jenkins server is diabled for this tutorial. You can, however trigger any of the preconfigured tutorial jobs created for this tutorial by making a POST request to the job endpoint. For example \n",
    "\n",
    "```\n",
    "# The base path to your jenkins host's remote job commands\n",
    "JENKINS_JOB_URL=\"https://<username>.sc18.training.agaveplatform.org/jenkins/job\"\n",
    "\n",
    "# The name of your Jenkins job as defined in the UI\n",
    "JEKNINS_JOB_NAME=\"funwave-build-pipeline\"\n",
    "\n",
    "# The unique build token assigned to your job. By default, we configured them\n",
    "# the same for this tutorial. You would obviously change this for external use\n",
    "BUILD_TOKEN=\"sc18-training-job\"\n",
    "\n",
    "# Manually trigger the funwave-build-pipeline job\n",
    "curl -sk \"$JENKINS_JOB_URL/$JEKNINS_JOB_NAME/build?token=$BUILD_TOKEN\"\n",
    "``` \n",
    "\n",
    "The Jenkins CLI is also disabled by default. While you can enable it manually from the Jenkins administration page, during the training the firewall on your host will not allow ssh access to your Jenkins container from the outside world, thus it is prefereable to use the web interface and remote build approach above. \n",
    "\n",
    "All scheduled builds should use polling rather than Github webhooks to trigger new jobs. This guarantees consistency between the tutorial material presented live when you have a publicly accessible IP address, and offline when you do not.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logging In\n",
    "\n",
    "We have already configured resources for you to use in this tutorial. \n",
    "\n",
    "### Virtual Machine\n",
    "\n",
    "Each of you have a dedicated VM provided by the [Jetstream Cloud](https://jetstream-cloud.org/). You will use this VM for the duration of the tutorial. \n",
    "\n",
    "### Training Account\n",
    "\n",
    "A training account on the Agave Platform's public tenant has also been allocated to you.\n",
    "\n",
    "\n",
    "### Login\n",
    "\n",
    "Your Jupyter server is available at `<username>.sc18.training.agaveplatform.org`. \n",
    "\n",
    "Your Jenkins server is available at `<username>.sc18.training.agaveplatform.org/jenkins`. \n",
    "\n",
    "Usernames will be training001 to training100. We will count off to determine our instance.\n",
    "\n",
    "When you first login, you will find it empty, save for a notebook named [INSTALL.ipynb](INSTALL.ipynb). Open this notebook by clicking on the notebook name, then click the *\"run\"* button. This will fetch all the tutorial notebooks from the tutorial's git repository an add them to your workspace. \n",
    "\n",
    "Once complete, open the [Config](Config.ipynb) notebook to being the meat of our tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Following along at home\n",
    "\n",
    "If you are following along with this tutorial at home, you can recreate the tutorial Jupyter server and sandbox environments by running the containers on your own server using the <a href=\"docker-compose.yml\">Docker Compose file.</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the above, you need to first set the environment variables `AGAVE_USERNAME`, `AGAVE_PASSWORD`, and `NGROK_TOKEN`. The first two should be your agave username and password as obtained from [the Agave registration page](https://public.agaveapi.co/create_account). The ngrok token should be obtained from [ngrok](https://ngrok.com)\n",
    "\n",
    "Ngrok will provide tunnelling for you so that Agave can ssh into your laptop or desktop machine. It will do this by setting the `VM_IPADDRESS`, `VM_HOSTNAME` and `VM_SSH_PORT` for you.\n",
    "\n",
    "Once you have these things setup, you should be able to run `docker-compose up -d` (note: you should run this command from the base directory of this repository.) You should then be able use your browser to connect to the tutorial setup on port 80 of your Docker host. If you are running Docker for Mac, Docker for Windows, or Docker CE on Linux, your address should be: (http://localhost)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}