Skip to content

DNA footprinting

Jump to bottom
twang15 edited this page Apr 1, 2021 · 9 revisions

How to set up Jupyter notebook on a cluster

  1. Login to a SCG node: ssh [email protected]
  2. Find out the IP address of the node you are on with the following command:
  • ifconfig | grep "inet\b " | sed -n "2p" | awk '{print $2}' gives me 171.67.92.56
  1. On the cluster node, run jupyter notebook --no-browser --port=8848 &
  • 8848 is the port for Jupiter notebook to listen (expecting request). You can pick a new one unoccupied by other service.
  1. On you laptop, run ssh -N -f -L 8848:localhost:8848 [email protected] to establish the ssh tunnel between your laptop and the cluster node
  2. On you laptop, enter http://localhost:8848/tree to your browser to send a Jupyter notebook request to the local port 8848, which will be tunneled by ssh on its port 22 to the remote cluster's 8848 port, where the Jupyter notebook server is listening.
  3. step 4 will pop up a webpage asking for token, which could be retrieved from the output of step 2. After inputing the token, you will be able to use remote jupyter notebook

Running a Jupyter notebook on a Slurm Cluster

Enable R in Jupyter notebook on SCG

  1. use pyenv to manage different python versions
  2. check the pip [taowang9@smsh11dsu-srcf-d15-36 ~]$ which pip alias pip='pip3' ~/.pyenv/shims/pip3
  3. install software
  • pip install jupyter
  • module load R/4.0.3
  • module load anaconda/3_5.0.1_20180125
  • conda install -c r r-irkernel
  • R #run a R session
    • install.packages('IRkernel')
    • IRkernel::installspec(user = TRUE)

Install R kernel for Jupyter

  1. wget https://cran.rstudio.com/src/base/R-4/R-4.0.5.tar.gz tar xvf R-4.0.5.tar.gz mkdir $HOME/tools wget https://sourceforge.net/projects/pcre/files/pcre2/10.36/pcre2-10.36.tar.gz tar xvf pcre2-10.36.tar.gz cd pcre2-10.36 ./configure --prefix=$HOME/tools make -j10 make install cd ../R-4.0.5 ./configure --prefix=$HOME/tools/ --with-pcre1=$HOME/tools/ make -j20 make install
  2. export PATH=$HOME/tools/bin:$PATH
  3. install IRkernel R install.packages('IRkernel') IRkernel::installspec()
  4. Ready to use it in Jupyter notebook

You may experience the following error:

[I 01:08:51.455 NotebookApp] Adapting from protocol version 5.0 (kernel 10b8352e-3c9c-4f6e-b4a5-2deb2d3370f5) to 5.3 (client). Error in .External2(C_X11, paste0("png::", filename), g$width, g$height, : unable to start device PNG Calls: ... evaluate -> dev.new -> do.call -> -> ok_device In addition: Warning message: In ok_device(filename, ...) : unable to open connection to X11 display '' Execution halted

To resolve it,

  • disconnect your laptop from all sessions to a destiny cluster node (otherwise, xquartz may be confused where to forward output)
  • brew install xquartz
  • launch xquartz
  • open Application/terminal
  • source ~/.bashrc
  • scg
  • password
  • duo code
  • jupyter notebook --no-browser --port=8848
  • in another new terminal (either Xquartz xterm or standalone iTerm2), ssh -L 8848:localhost:8848 [email protected]

Maybe open a new i-term can also solve this problem.

On-demand Desktop

  1. Install VPN: https://uit.stanford.edu/service/vpn
  2. Login VPN with SUNET ID and password, Duo authentication code
  3. Go to https://ondemand.scg.stanford.edu
  4. Select Jupyter notebook / IGV / Studio / etc...

DNA foot printing

DNA-seq

Atac-seq

Clone this wiki locally