Just for having fun learning and using locally for internet site analyses.
-
Yay! You're on Rails!
-
Now the scraper and various analyses and wiring it all into a single page.
$ gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
gpg: key 105BD0E739499BDB: public key "Piotr Kuczynski <[email protected]>" imported
gpg: key 3804BB82D39DC0E3: public key "Michal Papis (RVM signing) <[email protected]>" imported
gpg: Total number processed: 2
gpg: imported: 2
$ curl -sSL https://get.rvm.io | bash -s stable --ruby
$ source /home/[username]/.rvm/scripts/rvm
$ rvm --version
rvm 1.29.12 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
$ rvm list known
$ rvm --default use ruby-3.0.0
Using /home/[user]/.rvm/gems/ruby-3.0.0
$ curl -sL https://deb.nodesource.com/setup_15.x | sudo bash -
$ sudo apt install -y nodejs
$ node -v
If you don't have them already, you'll also have to install
$ sudo apt install gcc g++ make
$ gem update --system
$ gem -v
Install Ruby on Rails
$ gem install rails -v 6.1.3.1
$ rails -v
Postgres:
$ sudo apt install postgresql postgresql-contrib libpq-dev -y
$ sudo systemctl start postgresql
$ sudo systemctl enable postgresql
$ dpkg --status postgresql
Log in as sudo on PostgreSQL
$ sudo -u postgres psql
Change the postgres password
postgress=# \password postgres
Create role
postgress=# create role afander with createdb login password 'whateverpassword';
Check
postgress=# \du
Leave with Ctrl-Z
$ rails new afander -d postgresql
Correct the gemfile and
$ bundle install
In the config directory find file database.yml and configure development and test (two spaces in front of each line!)
development:
<<: *default
database: afander_development
username: afander
password: [whateverpasswordyouchose]
host: localhost
port: 5432
test:
<<: *default
database: afander_test
username: afander
password: [whateverpasswordyouchose]
host: localhost
port: 5432
Create the database
$ rails db:setup
Created database 'afander_development'
Created database 'afander_test'
...
Migrate the database
$ rails db:migrate
Install yarn if you don't already have it
$ curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
$ echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
$ sudo apt-get update
$ sudo apt install --no-install-recommends yarn
$ rails webpacker:install
And try to resolve all dependencies with yarn.
Launch
$ rails s -b localhost -p 80
$ gem install kimurai
Install x virtual framebuffer. It performs all graphical operations in virtual memory without showing any screen output.
$ sudo apt install -q -y xvfb
§ gem install selenium-webdriver
Install Chrome, Firefox and PhantomJS with drivers. Check and install the specific driver versions for Chrome (in my case version 89). For firefox I just used the latest geckodriver for now (until I get errors, heh).
Firefox
$ apt install firefox-esr
$ wget https://github.com/mozilla/geckodriver/releases/download/v0.29.1/geckodriver-v0.29.1-linux64.tar.gz
$ sudo tar -xvzf geckodriver-v0.29.1-linux64.tar.gz -C /usr/local/bin
Chrome
$ sudo wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
$ sudo apt install ./google-chrome-stable_current_amd64.deb
$ wget https://chromedriver.storage.googleapis.com/89.0.4389.23/chromedriver_linux64.zip
$ sudo unzip chromedriver_linux64.zip -d /usr/local/bin
$ sudo chown root:root /usr/local/bin/chromedriver
$ sudo chmod +x /usr/local/bin/chromedriver
Phantom
$ sudo apt install -q -y chrpath libxft-dev libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev
$ wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
$ tar -xvjf phantomjs-2.1.1-linux-x86_64.tar.bz2
$ sudo mv phantomjs-2.1.1-linux-x86_64 /usr/local/lib
$ sudo ln -s /usr/local/lib/phantomjs-2.1.1-linux-x86_64/bin/phantomjs /usr/local/bin
This project welcomes contributions and suggestions.