Skip to content

Installation AWS

Jump to bottom
catubc edited this page Mar 17, 2020 · 3 revisions

Installation - AWS

YASS has been tested and run using AWS EC2 services. The steps require setting up an AWS paid-services account, starting a YASS instance, uploading the data to the AWS servers, running YASS, downloading the spike-train results and terminating the AWS service.

Setup and log into AWS services

  1. Log into AWS services: https://aws.amazon.com/ec2/

  2. Make a paid account (need to enter credit card information as the required resources are pay-only services).

  3. [Optional] Future services using NCAP may be available where costs for first X hours will be covered by our research group. Please contact us to get more information.

Create an EC2 instance of YASS

  1. Start an EC2 instance from the AWS EC2 service by clicking on "EC2 – Services".

  2. Click on "Launch Instance".

  3. Search for AMI repository under the name “yass”.

  4. Select yass from community AMIs.

  5. Select EC2 environment. We recommend g4dn.12xlarge as it has been tested extensively (please refer to AWS for the hourly service cost for these nodes).

  6. Click "Review and Launch".

  7. Create a key pair for later use to transfer datafiles to AWS. • Select “Create a new key pair” • provide name (e.g. “yass”)

  • Take note of where the KEY.pem was saved.

Launch YASS instance

  1. Click on “View Instances”.

  2. Select Instance ID (Blue dot in first column) and click on "Connect"

  3. Select “EC2 Instance Connect” (Note: it may take a few minutes for the instance to be setup).

Alternatively use ssh + key to login. For example, for key "yass.pem" and user "ubuntu"

ssh -i "yass.pem" [email protected]

  1. (optional) Mount additional storage for processing. When using 4dn.12xlarge can use following commands:

sudo mkfs.ext4 -E nodiscard /dev/nvme1n1

sudo mount /dev/nvme1n1 /data

sudo chown -R ubuntu /data # changes the owner of the folder /data

Make run directory and edit config.yaml file:

  1. Make a root directory for running yass. For example: cd ENTER

    mkdir data

    cd data

  2. Download the default config.yaml and update it with your recordings information. We recommend downloading config.yaml file and editing it on your local machine before uploading (unless you are familiar with command line text editors).

  • add the root location of binary file and geometry
  • change sampling rates and # of channels
  • make geometry file (2 column x-y) text file (see sample here: geom.txt).

Upload raw data and config files directory and edit config.yaml file:

  1. Option 1 Secure Copy (scp). Copy raw data, config files and geometry from local drive to AWS instance using secure copy (scp).
  • Open terminal window ON HOME COMPUTER (not AWS side) and use scp command:

scp -i path_to_pem_secure_key path_to_local_filename [email protected]:/path_of_upload

Example using scp to upload data, geometry and config files:

scp -i /home/cat/Downloads/cat.pem /home/cat/data/data.bin [email protected]:data

scp -i /home/cat/Downloads/cat.pem /home/cat/data/config.yaml [email protected]:data

scp -i /home/cat/Downloads/cat.pem /home/cat/data/geom.txt [email protected]:data

  1. Option 2: sftp: Copy data using sftp (e.g. filezilla.com). See youtube instructions here: Filezille to AWS

Activate yass, test and run on new datasets:

  1. Activate conda environment for yass on instance: source activate yass

  2. [Optional]: Test yass installation:
    • navigate to yass/samples/10chan
    • delete any previous runs using “rm -rf tmp”
    • run:
    yass sort config.yaml

  3. Run yass on newly uploaded data:
    • navigate to newly uploaded data location • run yass sort config.yaml

Download results and close AWS instance:

  1. Location of yass output is data/tmp/spike_train.npy

  2. Optionally, dynamic templates are also available (i.e. time varying templates): /tmp/final_deconv/deconv_with_updates/templates_*.npy

  3. Download data instructions, same as copy to AWS, but with host/client exchanged (option: can use sftp service such as Filezilla).

Examples of downloading spike trains and template files:

`scp -i /home/cat/Downloads/cat_home.pem [email protected]:data/tmp/spike_train.npy /home/cat/Downloads/spike_train.npy`

`scp -i /home/cat/Downloads/cat_home.pem [email protected]:yass/samples/10chan/tmp/final_deconv/deconv_with_updates/templates/* /home/cat/Downloads/`
  1. Closing instance of AWS (or will be charged while service instance is open):
    • Return to webrowser AWS instance screen
    • click blue box of instance (first column)
    • Actions→ Instance State→ Terminate
Clone this wiki locally