diff --git a/docs/src/index.md b/docs/src/index.md index 01340a190..c5eaf0338 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -1,38 +1,38 @@ # Spyglass -**Spyglass** is a data analysis framework that facilitates the -storage, analysis, and sharing of neuroscience data to support -reproducible research. It is designed to be interoperable with the NWB -format and integrates open-source tools into a coherent framework. +**Spyglass** is a data analysis framework that facilitates the storage, +analysis, and sharing of neuroscience data to support reproducible research. It +is designed to be interoperable with the NWB format and integrates open-source +tools into a coherent framework. ## Getting Started This site hosts both [installation instructions](./installation.md) and -[tutorials](./notebooks/index.md) to help you get started with Spyglass. -We recommend running the notebooks yourself. They can be downloaded from -GitHub [here](https://github.com/LorenFrankLab/spyglass). +[tutorials](./notebooks/index.md) to help you get started with Spyglass. We +recommend running the notebooks yourself. They can be downloaded from GitHub +[here](https://github.com/LorenFrankLab/spyglass). ## Diving Deeper -The [API Reference](./api/index.md) provides a detailed description of all -the tables and class functions in Spyglass via python docstrings. Potential +The [API Reference](./api/index.md) provides a detailed description of all the +tables and class functions in Spyglass via python docstrings. Potential contributors should also read the [Developer Guide](./contribute.md). Those interested in in hosting a Spyglass instance for their own data should read the [database management guide](./misc/database_management.md). We have a series of additional docs under the [misc](./misc/index.md) folder -that may be helpful. Our [changelog](./CHANGELOG.md) highlights the changes -that have been made to Spyglass over time and the [copyright](./copyright.md) -page contains license information. +that may be helpful. Our [changelog](./CHANGELOG.md) highlights the changes that +have been made to Spyglass over time and the [copyright](./LICENSE.md) page +contains license information. ## Citing Spyglass -Kyu Hyun Lee, Eric Denovellis, Ryan Ly, Jeremy Magland, Jeff Soules, -Alison Comrie, Jennifer Guidera, Rhino Nevers, Daniel Gramling, Philip -Adenekan, Ji Hyun Bak, Emily Monroe, Andrew Tritt, Oliver Rübel, Thinh -Nguyen, Dimitri Yatsenko, Joshua Chu, Caleb Kemere, Samuel Garcia, -Alessio Buccino, Emily Aery Jones, Lisa Giocomo, and Loren Frank. -'Spyglass: A Data Analysis Framework for Reproducible and Shareable -Neuroscience Research.' (2022) Society for Neuroscience, San Diego, CA. +Kyu Hyun Lee, Eric Denovellis, Ryan Ly, Jeremy Magland, Jeff Soules, Alison +Comrie, Jennifer Guidera, Rhino Nevers, Daniel Gramling, Philip Adenekan, Ji +Hyun Bak, Emily Monroe, Andrew Tritt, Oliver Rübel, Thinh Nguyen, Dimitri +Yatsenko, Joshua Chu, Caleb Kemere, Samuel Garcia, Alessio Buccino, Emily Aery +Jones, Lisa Giocomo, and Loren Frank. 'Spyglass: A Data Analysis Framework for +Reproducible and Shareable Neuroscience Research.' (2022) Society for +Neuroscience, San Diego, CA. diff --git a/docs/src/misc/database_management.md b/docs/src/misc/database_management.md index 2971318af..dfcfa7eeb 100644 --- a/docs/src/misc/database_management.md +++ b/docs/src/misc/database_management.md @@ -69,7 +69,147 @@ dj.set_password() ## Database Backups -Coming soon... +The following codeblockes are a series of files used to back up our database and +migrate the contents to another server. Some conventions to note: + +- `.host`: files used in the host's context +- `.container`: files used inside the database Docker container +- `.env`: files used to set environment variables used by the scripts for + database name, backup name, and backup credentials + +This backup process uses a dedicated backup user, that an admin would need to +criate with the relevant permissions. + +### mysql.env.host + +
+MySQL host environment variables + +Values may be adjusted as needed for different building images. + +```bash +ROOT_PATH=/usr/local/containers/mysql # path to this container's working area + +# variables for building image +SRC=ubuntu +VER=20.04 +DOCKERFILE=Dockerfile.base + +# variables for referencing image +IMAGE=mysql8 +TAG=u20 +# variables for running the container +CNAME=mysql-datajoint +MACADDR=4e:b0:3d:42:e0:70 +RPORT=3306 + +# variables for initializing/relaunching the container +# - where the mysql data and backups will live - these values +# are examples +DB_PATH=/data/db +DB_DATA=mysql +DB_BACKUP=/data/mysql-backups + +# backup info +BACK_USER=mysql-backup +BACK_PW={password} +BACK_DBNAME={database} +# mysql root password - make sure to remove this AFTER the container +# is initialized - and this file will be replicated inside the container +# on initialization, so remove it from there: /opt/bin/mysql.env +``` + +
+ +### backup-database.sh.host + +This script runs the mysql-backup container script (exec inside the container) +that dumps the database contents for each database as well as the entire +database. Use cron to set this to run on your desired schedule. + +
+MySQL host docker exec + +```bash +#!/bin/bash + +PRIOR_DIR=$(pwd) +cd /usr/local/containers/mysql || exit +. mysql.env +cd "$(dirname ${ROOT_PATH})" +# +docker exec ${CNAME} /opt/bin/mysql-backup.csh +# +cd "$(dirname ${DB_BACKUP})" +# +cd ${PRIOR_DIR} +``` + +
+ +### mysql-backup-xfer.csh.host + +This script transfers the backup to another server 'X' and is specific for us as +it uses passwordless ssh keys to a local unprivileged user on X that has the +mysql backup area on X as that user's home. + +
+MySQL host transfer script + +```bash +#!/bin/csh +set td=`date +"%Y%m%d"` +cd /data/mysql-backups +scp -P {port} -i ~/mysql-backup -r ${database}-${td} mysql-backup@${X}:~/ +/bin/rm -r lmf-db-${td} +``` + +
+ +### myenv.csh.container + +
+Docker container environment variables + +```bash +set db_backup=mysql-backups +set back_user=mysql-backup +set back_pw={password} +set back_dbname={database} +``` + +
+ +### mysql-backup.csh.container + +
+Generate backups from within container + +```bash +#!/bin/csh +source /opt/bin/myenv.csh +set td=`date +"%Y%m%d"` +cd /${db_backup} +mkdir ${back_dbname}-${td} + +set list=`echo "show databases;" | mysql --user=${back_user} --password=${back_pw}` +set cnt=0 + +foreach db ($list) + if ($cnt == 0) then + echo "dumping mysql databases on $td" + else + echo "dumping MySQL database : $db" + # Per-schema backups + mysqldump $db --max_allowed_packet=512M --user=${back_user} --password=${back_pw} > /${db_backup}/${back_dbname}-${td}/mysql.${db}.sql + endif +@ cnt = $cnt + 1 +end +# Full database backup +mysqldump --all-databases --max_allowed_packet=512M --user=${back_user} --password=${back_pw} > /${db_backup}/${back_dbname}-${td}/mysql-all.sql +``` + +
## File Cleanup diff --git a/src/spyglass/position/v1/position_dlc_centroid.py b/src/spyglass/position/v1/position_dlc_centroid.py index 62650a9e9..351f577aa 100644 --- a/src/spyglass/position/v1/position_dlc_centroid.py +++ b/src/spyglass/position/v1/position_dlc_centroid.py @@ -467,8 +467,7 @@ def fetch1_dataframe(self): def four_led_centroid(pos_df: pd.DataFrame, **params): - """ - Determines the centroid of 4 LEDS on an implant LED ring. + """Determines the centroid of 4 LEDS on an implant LED ring. Assumed to be the Green LED, and 3 red LEDs called: redLED_C, redLED_L, redLED_R By default, uses (greenled + redLED_C) / 2 to calculate centroid If Green LED is NaN, but red center LED is not,