-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
jupyterhub is a fully integrated service on euler now
- Loading branch information
1 parent
12c3190
commit 0ef5f27
Showing
1 changed file
with
27 additions
and
129 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -73,12 +73,14 @@ | |
| "\n", | ||
| "### 1. Load dependencies\n", | ||
| "\n", | ||
| "Use the new software stack. Unless you have already done so, run `set_software_stack.sh new`.\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "env2lmod\n", | ||
| "module load gcc/8.2.0 python/3.9.9 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.22.1 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n", | ||
| "module load gcc/8.2.0 python/3.10.4 hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n", | ||
| "```\n", | ||
| "\n", | ||
| "You need to execute these two lines every time you login to Euler before Climada can be used. To safe yourself from doing it manually, one can append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler." | ||
| "You need to execute this every time you login to Euler before Climada can be used. \n", | ||
| "To safe yourself from doing it manually, append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler." | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -91,9 +93,11 @@ | |
| "```bash\n", | ||
| "python -c 'import climada; print(climada.__file__)'\n", | ||
| "```\n", | ||
| " should output something like this:\n", | ||
| "\n", | ||
| "should output something like this:\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "/cluster/apps/nss/gcc-8.2.0/python/3.9.9/x86_64/lib64/python3.9/site-packages/climada/__init__.py\n", | ||
| "/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86_64/lib64/python3.10/site-packages/climada/__init__.py\n", | ||
| "```" | ||
| ] | ||
| }, | ||
|
|
@@ -419,162 +423,56 @@ | |
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "It is possible to run a Jupyter Notebook on Euler and access it through a browser from your local machine.\n", | ||
| "To make this possible you have to do some preparation steps on both, your local machine and on Euler.\n", | ||
| "\n", | ||
| "**Windows Users:** to run the commands given below on your local computer you have to use `git-bash` (or WSL `bash` if Ubuntu on Windows is installed)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "\n", | ||
| "### 1. Get the _Jupyter on Euler_ scripts and documentation\n", | ||
| "on your local computer, run these commands:\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "git clone [email protected]:sfux/Jupyter-on-Euler-or-Leonhard-Open.git\n", | ||
| "cd Jupyter-on-Euler-or-Leonhard-Open\n", | ||
| "git checkout add_slurm\n", | ||
| "```" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "\n", | ||
| "### 2. Create an SSH key pair\n", | ||
| "It is possible to run a Jupyter Notebook on Euler within a JupytherHub instance running as an interactive slurm job.\n", | ||
| "See the documentation https://scicomp.ethz.ch/wiki/JupyterHub.\n", | ||
| "\n", | ||
| "on your local computer, run this command:\n", | ||
| "For using climada inside the jupyter notebook, create a `.jupyterlabrc` file in your Euler home directory with the following content:\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "ssh-keygen ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler\n", | ||
| "```\n", | ||
| "\n", | ||
| "When prompted, enter a _strong_ passphrase.\n", | ||
| "\n", | ||
| "Then, run\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "ssh-add $HOME/.ssh/id_ed25519_euler\n", | ||
| "module purge\n", | ||
| "module load StdEnv gcc/8.2.0 python_gpu/3.10.4 eth_proxy r/4.2.2 julia/1.8.5 hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n", | ||
| "```\n", | ||
| "\n", | ||
| "and provide the passphrase from above.\n", | ||
| "\n", | ||
| "**Windows Users:** `ssh-add` is working together with an SSH-Agent, which is not running out of the box in `git-bash`. Follow the instructions [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/working-with-ssh-key-passphrases) to make it available." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "\n", | ||
| "### 3. Copy the public key into the authorized_keys file on Euler\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "ssh-copy-id -i $HOME/.ssh/id_ed25519_euler.pub [email protected]\n", | ||
| "```\n", | ||
| "\n", | ||
| "replace `USERNAME` with your ETH user name. When prompted to enter the passwort provide your LADP password (i.g., it is the same as your E-mail password).\n", | ||
| "\n", | ||
| "At this point you should be able to access Euler through `ssh` without password:\n", | ||
| "\n", | ||
| "```bash\n", | ||
| "ssh [email protected]\n", | ||
| "```\n", | ||
| "should bring you directly to the Euler cluster." | ||
| "Then login to https://jupyter.euler.hpc.ethz.ch/ and start a JupyterLab server." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "\n", | ||
| "### 4. Prepare Euler environment\n", | ||
| "### Using a virtual environment in a Jupyter notebeook\n", | ||
| "\n", | ||
| "It is highly recommanded to use a python environment and a github clone as described above under **Working with Git Branches**.\n", | ||
| "By default the pre-installed climada version is running in your notebooks. If you want to use climada from source you can simply install a python kernel from the `climada_venv` environment, see [Working with Git branches](#Working-with-Git-branches)\n", | ||
| "\n", | ||
| "Login to Euler.\n", | ||
| "\n", | ||
| "Unless you have done so already you need to permanently switch to the new software stack:\n", | ||
| "```bash\n", | ||
| "set_software_stack.sh new\n", | ||
| "```\n", | ||
| "\n", | ||
| "Unless you have done so already you need to append the default module selection in your `~/.bashrc` file:\n", | ||
| "```bash\n", | ||
| "echo >> ~/.bashrc\n", | ||
| "echo module load gcc/8.2.0 python/3.9.9 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.22.1 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5 >> ~/.bashrc\n", | ||
| "```\n", | ||
| "Install an IPyhton-kernel:\n", | ||
| "\n", | ||
| "Activate your python environment:\n", | ||
| "```bash\n", | ||
| "source /cluster/project/climate/$USER/climada_venv/bin/activate\n", | ||
| "```\n", | ||
| "\n", | ||
| "Install an IPyhton-kernel:\n", | ||
| "```\n", | ||
| "python -m ipykernel install --user --name climada_venv\n", | ||
| "``` \n", | ||
| "\n", | ||
| "Logout." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "\n", | ||
| "### 5. Adjust the script\n", | ||
| "\n", | ||
| "Open the script `start_jupyter_nb.sh` in an editor and change the gcc and python version numbers in lines 421/422 so they look like this:\n", | ||
| "```\n", | ||
| "[421] JNB_MODULE_COMMAND=\"gcc/8.2.0 r/4.0.2 python/3.9.9 eth_proxy $JNB_JULIA\"\n", | ||
| "[422] echo -e \"Using new software stack (gcc/8.2.0 python/3.9.9 eth_proxy $JNB_JULIA)\"\n", | ||
| "\n", | ||
| "```\n", | ||
| "\n", | ||
| "make the script executable:\n", | ||
| "```bash\n", | ||
| "chmod +x start_jupyter_nb.sh\n", | ||
| "```" | ||
| "Start a new JupyterLab server, the `climada_venv` kernel should appear in the list of available kernels in JupyterHub." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "\n", | ||
| "### 6. Start a Jupyter Notebook on Euler\n", | ||
| "\n", | ||
| "At this point everything is ready. You should be able to start a Jupter Notebook remotely running on Euler.\n", | ||
| "## Trouble shooting\n", | ||
| "\n", | ||
| "Check out the script:\n", | ||
| "```bash\n", | ||
| "./start_jupyter_nb.sh\n", | ||
| "```\n", | ||
| "### 1. Python Module not found or available\n", | ||
| "\n", | ||
| "Run Jupyter on Euler on a single cpu (default) with 1G of memory (default) in your projects directory with the slurm batch system:\n", | ||
| "```bash\n", | ||
| "./start_jupyter_nb.sh -u NETHZID -b SLURM -w /cluster/project/climate/USERNAME\n", | ||
| "```\n", | ||
| "- Make sure your python environment is activated.\n", | ||
| "- Run `pip install --upgrade MISSING_MODULE`.\n", | ||
| "\n", | ||
| "If all went fine your default browser will be opened after some time with Jupyter running in your project directory.\\\n", | ||
| "When opening or creating a notebook make sure you select the `climada_venv` kernel." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 2. Upgrading from Python 3.9 to 3.10\n", | ||
| "\n", | ||
| "## Trouble shooting\n", | ||
| "Virtual environments created with Python 3.9 are i.g. not working for Python 3.10.\n", | ||
| "In particular Python kernels from 3.9 environments will fail to connect in a Jupyter notebook on https://jupyter.euler.hpc.ethz.ch/.\n", | ||
| "\n", | ||
| "### 1. Python Module not found or available\n", | ||
| "- Make sure your python environment is activated.\n", | ||
| "- Run `pip install --upgrade MISSING_MODULE`." | ||
| "- It's suggested to create new environments and remove the old kernels from `~/.local/share/jupyter/kernels/`." | ||
| ] | ||
| } | ||
| ], | ||
|
|
||