Huge refactor. Rename to serf, better distribution, bring in functi…

…ons from Guillaume

INSTALL_MYSQL.md

@@ -0,0 +1,83 @@
+## Installing MySQL Database and Python connector
+
+We assume that you followed other instructions to install Python and Django. The rest of the instructions are for installing the MySQL database server and its interfaces. Though any DBMS should work, I use MySQL because some of the data I work with were originally recorded into MySQL.
+
+### On OS X
+
+1. Install [homebrew](http://brew.sh/) if you don't have it already.
+1. `brew install mysql`
+1. Optional - Change default data dir
+    * `mkdir /path/to/datadir`
+    * `sudo mysqld --initialize-insecure --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/path/to/datadir`
+    * You may need to `chown -R mysql:wheel /path/to/datadir` and `chmod -R 755 /path/to/datadir`
+    * Add a `datadir` line to my.cnf (see below))
+1. Edit ~/.profile and add the following lines.
+    `export PATH=/usr/local/mysql/bin:$PATH`
+    `export PATH=/usr/local/mysql/lib:$PATH`
+1. `source ~/.profile` or you may have to close and reopen terminal.
+1. Run the server `mysql.server start` or `mysqld_safe &`
+1. Your Python environment will require some additional packages. e.g.: `conda install mysqlclient openssl`
+
+### On Linux
+
+First we'll install and configure MySQL server.
+1. [Follow these instructions](https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-18-04)
+1. Optional - [Change default datadir](https://www.digitalocean.com/community/tutorials/how-to-move-a-mysql-data-directory-to-a-new-location-on-ubuntu-18-04)
+1. `conda install mysqlclient openssl=1.0.2r`  <-- haven't tested the openssl version requirement in a long time.
+
+### On Windows
+
+1. Install [MySQL Community Server](https://dev.mysql.com/downloads/mysql/)
+    * Choose the Developer package (MySQL Server, Workbench, and Shell)
+    * Work through pre-requisites
+    * MySQL80 will be your service name unless you change this
+1. `conda install mysqlclient openssl`
+
+## Configuring the database server for use with serf
+
+The MySQL server will need to have some minimal configuration done to it. The easiest way to do this is to open MySQL Workbench with Administrator privileges (admin needed if mysql ini file stored system directory).
+You can also edit the settings file manually. Possible locations:
+* /etc/my.cnf
+* /etc/mysql/my.cnf
+* /usr/local/etc/my.cnf
+* ~/.my.cnf
+* C:\ProgramData\MySQL\MySQL Server 8.0\my.ini
+
+You'll likely want to change the data directory. In the past I've also needed to change from the default storage engine (InnoDB) to the MyISAM storage engine because I had trouble getting good performance out of InnoDB when working with some old rat data. Here are some changes I've made in the past:
+
+```
+[mysqld]
+datadir = /Volumes/STORE/eerfdata
+default-storage-engine = MyISAM
+default_tmp_storage_engine = MyISAM
+query_cache_type = 1
+key_buffer_size = 2G
+query_cache_limit = 400M
+```
+
+If you changed the datadir then you'll need to copy the original data structure to the new location (Windows) or use `sudo mysql_install_db --user=root --defaults-file=/etc/my.cnf`.
+
+Then restart the server: `mysqld_safe & --defaults-file=/etc/my.cnf`
+
+It is not necessary to specify the defaults file when using the default location (/etc/my.cnf).
+
+1. Create the root database for the application:
+    - Mac/Linux: 
+        - `mysql -uroot -p`
+        - `create database serf character set utf8;`
+        - `exit;`
+    - On Windows (Workbench GUI):
+      - Open Workbench, connect to database (Database > Connect to Database)
+      - Enter your username and password
+      - Make a new schema called `serf`
+      - [Refer to this tutorial for a visual guide](https://www.mysqltutorial.org/mysql-create-database/)
+
+You are now ready to install the database for the serf app.
+
+## Additional Tips for installing MySQL
+
+1. Optional (mandatory for Matlab ORM): Install additional SQL triggers to
+    - Automatically log a new entry or a change to subject_detail_value
+    - Automatically set the stop_time field of a datum to +1s for trials or +1 day for days/periods.
+    - Automatically set the number of a new datum to be the next integer greater than the latest for that subject/span_type.
+    - After installing the databases for the serf app, from shell/terminal, run `mysql -uroot mysite < serf.sql`

django-eerf/README.md → MISC.md

@@ -1,76 +1,16 @@
-=====
-EERFAPP
-=====
-
-eerfapp is a Django app. The core of the app is its database. The database schema is designed to provide a complete and flexible representation of epoched neurophysiological data.
-
-![Database Schema](/models.png?raw=true "Database Schema")
-
-# Setup Instructions
-Before proceeding, consult our guide for [Installing Python and Django and Databases](https://github.com/cboulay/EERF/blob/master/INSTALL.md).
-Ensure that you have already run `python setup.py install` in the `eerf/django-eerf`.
-  - From `eerf/django-eerf/build/lib`, copy `eerfapp` and `eerfhelper` folders into your python environment's `site-packages` folder.
-
-## Installing the Django EERF Web Application     
-1. Navigate inside your expdb project
-  - If you haven't created it already, `django-admin startproject expdb`
-1. Include the eerfapp URL configurations in your project's `urls.py`:
-  - Add this to your import statements `from django.conf.urls import url, include`
-  - Add the following under `urlpatterns` list:
-    - `url(r'^eerfapp/', include(('eerfapp.urls','eerfapp'), namespace="eerfapp")),`
-1. Open `settings.py` and apply the following changes:
-  - Under `INSTALLED_APPS` add `eerfapp,` to the list
-  - Double check that you have these changes under `DATABASES`:
-    - `'ENGINE': 'django.db.backends.mysql'`
-    - `'NAME': 'expdb'`
-    - Credentials defined here must be correct
-      - [Create a MySQL OPTIONS file](https://docs.djangoproject.com/en/2.2/ref/databases/#connecting-to-the-database) and assign an username and password
-
-1. Activate your conda environment (e.g. environment name sql)
-   - Linux/Mac: `source ~/miniconda3/bin/activate && conda activate sql`
-   - Windows: Open Anaconda Prompt and type `conda activate sql`
-1. Run `python manage.py makemigrations eerfapp` in your expdb project
-1. You should see the following output:
- ```
-Migrations for 'eerfapp':
-  /Users/mikkeyboi/miniconda3/envs/sql/lib/python3.6/site-packages/eerfapp/migrations/0001_initial.py
-    - Create model Datum
-    - Create model DatumFeatureValue
-    - Create model DetailType
-    - Create model FeatureType
-    - Create model Subject
-    - Create model System
-    - Create model DatumFeatureStore
-    - Create model DatumStore
-    - Create model SubjectLog
-    - Add field feature_type to datumfeaturevalue
-    - Add field subject to datum
-    - Add field trials to datum
-    - Create model SubjectDetailValue
-    - Alter unique_together for datumfeaturevalue (1 constraint(s))
-    - Create model DatumDetailValue
-    - Alter unique_together for datum (1 constraint(s))
-
- ```
-
-1. Run `python manage.py migrate`, and it should `Applying eerfapp.0001_initial... OK`
-1. Create a superuser for administrative control on the django side `python manage.py createsuperuser`
-   - Take note of these credentials. The username will default as your account name
-1. Test your server `python manage.py runserver` and go to `localhost:8000/admin`
-
-
 ## Developing in Django
-Django has a [good tutorial](https://docs.djangoproject.com/en/3.0/intro/tutorial01/) to familiarize you with the structure of the project you just built.
-To send commands to your server, run `python manage.py shell` in your expdb project. [Here are some commands you can try](https://docs.djangoproject.com/en/3.0/intro/tutorial02/#playing-with-the-api).
-Running commands like this in REPL may not be ideal for development. If you wish to run scripts that call and write to the database, you have to include the following in the script you wish to run:
- ``` python
+
+Django has a [good tutorial](https://docs.djangoproject.com/en/3.0/intro/tutorial01/) to familiarize you with the structure of the project you just built. To send commands to your server, run `python manage.py shell` in your expdb project. [Here are some commands you can try](https://docs.djangoproject.com/en/3.0/intro/tutorial02/#playing-with-the-api). Running commands like this in REPL may not be ideal for development. If you wish to run scripts that call and write to the database, you have to include the following in the script you wish to run:
+ 
+```python
 from django.core.management import execute_from_command_line
 os.environ['DJANGO_SETTINGS_MODULE'] = 'expdb.settings'
 execute_from_command_line()
- ```
+```
+
 Below is an example where I pass some multidimensional random data into the server, retrieve it, and compare the their values (sent as a blob, retrieved in bytes).
 
-``` python
+```python
 import os
 import pdb
 import numpy as np