Adventures in local web app instance installation

I recently (10/1/2019) installed eol_website on a GNU/Linux laptop (distro = PureOS which is similar to Ubuntu). Here is a cleaned up version of what I did. I may have made mistakes in the cleanup process, but maybe this information will be useful.

UPDATE 3/20/2020: Ruby version is now 2.6.5; added further webapp setup instructions

Install and set up mariadb (= mysql)

sudo apt-get install libmariadb-dev
sudo apt-get install mariadb-server
cp -p config/secrets.sample.yml config/secrets.tml
  <add json_web_token_secret: ... to the secrets file>
  <start mariadb = mysql>
sudo mysql_secure_installation
  <set the root password if you've forgotten it>
sudo mysql -u root -p
create user 'eol'@'localhost' identified by 'eol';
create database eol_development;
grant all privileges on eol_development.* to 'eol'@'localhost';
flush privileges;

Install and start neo4j, which requires Java

As of 3/20/2020, the EOL production server is running neo4j 3.5.7. I am gambling here that 3.5.11 is compatible, but to be careful us 3.5.7 (or whatever you know to be running at SI).

sudo apt-get install openjdk-8-jdk
  <download Neo4j Community Edition 3.5.11 from neo4j.com>
tar xzf ~/Downloads/neo4j-community-3.5.11-unix.tar.gz 
cd /usr/local/src; tar xzf ~/Downloads/neo4j-community-3.5.11-unix.tar.gz
/usr/local/src/neo4j-community-3.5.11/bin/neo4j start

I run neo4j in user mode, as it is not necessary to run it as root and running daemons in user mode is good security hygiene. I restart it manually after each reboot.

Install elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.1-amd64.deb
sudo dpkg -i elasticsearch-7.2.1-amd64.deb
sudo systemctl start elasticsearch

Clone the eol_website repository

Like it says. Then cd to the main directory of that clone.

Install appropriate version of ruby

sudo apt-get install rbenv
rbenv init

Add the following to your .bashrc: eval "$(rbenv init -)". Restart bash to activate this command.

The following is necessary only if rbenv install says it has never heard of the ruby version you asked it to install. (ruby-build has a hardwired list of available ruby versions and sometimes it gets out of date.)

mkdir -p "$(rbenv root)"/plugins
git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build

Now get the right version.

rbenv install 2.6.5
rbenv rehash

Configure the web app

Get its 'gems':

bin/bundle

(I had considerable trouble getting the currect version of bundle but it's my belief that that issue has been resolved.)

Copy config/secrets.sample.yml to config/secrets.yml and edit it as follows:

1. Under db:, set username: eol and password: eol
2. Under repository:, set url: to either https://beta-repo.eol.org or https://content.eol.org (I'm not sure which would be better for general development... beta is missing 1/3 of the resources, and is usually behind production, but sometimes it has experimental resource versions that need to be tested)
3. Set neo4j username and password if you changed them (see mariadb setup - suggestion there is user eol and password eol)
4. Under development: add json_web_token_secret: xxx where xxx is a random hex string (if your local instance is not on the Internet then doesn't much matter what you put there)

Set up web app database

bin/rake db:environment:set RAILS_ENV=development
bin/rake db:schema:load RAILS_ENV=development

N.b. db:schema:load is only for a fresh database. Use db:migrate under ordinary circumstances.

Start the web app

bin/rails s
  <get api token if you want to use web services>

For more durable server access do something like

nohup bin/rails s >rails.nohup &

Start to populate the database

bin/rails r "TraitBank::Admin.setup"
bin/rails r "ImportRun.delete_all" 
bin/rails r 'user = User.create!({email: "[email protected]", \
 username: "admin", password: "not-in-wiki", role: :admin, \
 age_confirm: 1, tou_confirm: 1}); user.activate'

(actually that user setup command doesn't work; no way to log in... TBD)

Get resource information and fix resource ids

bin/rake sync

At this point all the local resource ids (id column of the resources table) are essentially random. We would like for them to agree with the EOL resource repository that we synced with (specified in secrets.yml) so that we can load resources. To do this we need to get ahold of the correct ids, and alter the keys in the resources table in the database.

First, to ensure we don't have resource records that compete (even if transiently) for unique ids with the records that deserve them, modify all the ids so they don't conflict. At the mysql prompt, do this:

update resources set id = id + 10000;

Now get the correct ids from the publishing site you want to be consistent with: (note, not a harvesting/repository site)

wget "http://eol.org/resources.json?per_page=1000"

Finally, renumber the resource records using SQL UPDATE. The following script will do this (it deserves a better home)

wget https://github.com/jar398/testbed/blob/master/clobber_resource_ids.rb
bin/rails r clobber_resource_ids.rb

Load some resources

At the very least, you'll need the dynamic hierarchy. You have to get the resource id for each resource you want to 'publish' (TBD - not sure the best way to do this - search on content.eol.org/resources or something?). E.g. the following should load the dynamic hierarchy:

bin/rake publish ID=724

Hmm, I can't get this to work...

Misc

This might be useful:

bin/rails r TraitBank::Denormalizer.set_canonicals

If you want to work with some representative clade instead of the entire DH:

bin/rake clade:read CLADE_FILE=7662_data.tgz

except this doesn't seem to work.

Maybe in future for benefit of neo4j: increase OS limit on number of open files