Decoding pipeline edits for imported data

CHANGELOG.md

@@ -20,7 +20,7 @@
 ### Pipelines
 
 - Spike sorting: Add SpikeSorting V1 pipeline. #651
-- LFP: Minor fixes to LFPBandV1 populator. #706
+- LFP: Minor fixes to LFPBandV1 populator and `make`. #706, #795
 - Linearization:
     - Minor fixes to LinearizedPositionV1 pipeline #695
     - Rename `position_linearization` -> `linearization`. #717
@@ -34,6 +34,7 @@
     - Use the new `non_local_detector` package for decoding #731
     - Allow multiple spike waveform features for clusterelss decoding #731
     - Reorder notebooks #731
+    - Add fetch class functionality to `Merge` table. #783, #786
 
 ## [0.4.3] (November 7, 2023)
 

dj_local_conf_example.json

@@ -14,7 +14,7 @@
   "display.show_tuple_count": true,
   "database.use_tls": null,
   "enable_python_native_blobs": true,
-  "filepath_checksum_size_limit": null,
+  "filepath_checksum_size_limit": 1073741824,
   "stores": {
     "raw": {
       "protocol": "file",
@@ -28,14 +28,28 @@
     }
   },
   "custom": {
+    "debug_mode": "false",
+    "test_mode": "false",
     "spyglass_dirs": {
-      "base": "/your/path/like/stelmo/nwb/"
+      "base": "/your/base/path",
+      "raw": "/your/base/path/raw",
+      "analysis": "/your/base/path/analysis",
+      "recording": "/your/base/path/recording",
+      "sorting": "/your/base/path/spikesorting",
+      "waveforms": "/your/base/path/waveforms",
+      "temp": "/your/base/path/tmp",
+      "video": "/your/base/path/video"
     },
     "kachery_dirs": {
-      "cloud": "/your/path/.kachery-cloud"
+      "cloud": "/your/base/path/kachery_storage",
+      "storage": "/your/base/path/kachery_storage",
+      "temp": "/your/base/path/tmp"
     },
     "dlc_dirs": {
-      "base": "/your/path/like/nimbus/deeplabcut/"
+      "base": "/your/base/path/deeplabcut",
+      "project": "/your/base/path/deeplabcut/projects",
+      "video": "/your/base/path/deeplabcut/video",
+      "output": "/your/base/path/deeplabcut/output"
     },
     "kachery_zone": "franklab.default"
   }

docs/src/index.md

@@ -2,46 +2,49 @@
 
 ![Figure 1](./images/fig1.png)
 
-**Spyglass** is an open-source software framework designed to offer reliable
-and reproducible analysis of neuroscience data and sharing of the results
-with collaborators and the broader community.
+**Spyglass** is an open-source software framework designed to offer reliable and
+reproducible analysis of neuroscience data and sharing of the results with
+collaborators and the broader community.
 
 Features of Spyglass include:
 
-+ **Standardized data storage** - Spyglass uses the open-source
-  [Neurodata Without Borders: Neurophysiology (NWB:N)](https://www.nwb.org/)
-  format to ingest and store processed data. NWB:N is a standard set by the BRAIN
-  Initiative for neurophysiological data ([Rübel et al., 2022](https://doi.org/10.7554/elife.78362)).
-+ **Reproducible analysis** - Spyglass uses [DataJoint](https://datajoint.com/)
-  to ensure that all analysis is reproducible. DataJoint is a data management
-  system that automatically tracks dependencies between data and analysis code. This
-  ensures that all analysis is reproducible and that the results are
-  automatically updated when the data or analysis code changes.
-+ **Common analysis tools** - Spyglass provides easy usage of the open-source packages
-  [SpikeInterface](https://github.com/SpikeInterface/spikeinterface),
-  [Ghostipy](https://github.com/kemerelab/ghostipy), and [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut)
-  for common analysis tasks. These packages are well-documented and have active
-  developer communities.
-+ **Interactive data visualization** - Spyglass uses [figurl](https://github.com/flatironinstitute/figurl)
-  to create interactive data visualizations that can be shared with collaborators
-  and the broader community. These visualizations are hosted on the web
-  and can be viewed in any modern web browser. The interactivity allows users to
-  explore the data and analysis results in detail.
-+ **Sharing results** - Spyglass enables sharing of data and analysis results via
-  [Kachery](https://github.com/flatironinstitute/kachery-cloud), a
-  decentralized content addressable data sharing platform. Kachery Cloud allows
-  users to access the database and pull data and analysis results directly
-  to their local machine.
-+ **Pipeline versioning** - Processing and analysis of data in neuroscience is
-  often dynamic, requiring new features. Spyglass uses *Merge tables* to ensure that
-  analysis pipelines can be versioned. This allows users to easily use and compare
-  results from different versions of the analysis pipeline while retaining
-  the ability to access previously generated results.
-+ **Cautious Delete** - Spyglass uses a `cautious delete` feature to ensure
-  that data is not accidentally deleted by other users. When a user deletes data,
-  Spyglass will first check to see if the data belongs to another team of users.
-  This enables teams of users to work collaboratively on the same database without
-  worrying about accidentally deleting each other's data.
+- **Standardized data storage** - Spyglass uses the open-source
+    [Neurodata Without Borders: Neurophysiology (NWB:N)](https://www.nwb.org/)
+    format to ingest and store processed data. NWB:N is a standard set by the
+    BRAIN Initiative for neurophysiological data
+    ([Rübel et al., 2022](https://doi.org/10.7554/elife.78362)).
+- **Reproducible analysis** - Spyglass uses [DataJoint](https://datajoint.com/)
+    to ensure that all analysis is reproducible. DataJoint is a data management
+    system that automatically tracks dependencies between data and analysis
+    code. This ensures that all analysis is reproducible and that the results
+    are automatically updated when the data or analysis code changes.
+- **Common analysis tools** - Spyglass provides easy usage of the open-source
+    packages [SpikeInterface](https://github.com/SpikeInterface/spikeinterface),
+    [Ghostipy](https://github.com/kemerelab/ghostipy), and
+    [DeepLabCut](https://github.com/DeepLabCut/DeepLabCut) for common analysis
+    tasks. These packages are well-documented and have active developer
+    communities.
+- **Interactive data visualization** - Spyglass uses
+    [figurl](https://github.com/flatironinstitute/figurl) to create interactive
+    data visualizations that can be shared with collaborators and the broader
+    community. These visualizations are hosted on the web and can be viewed in
+    any modern web browser. The interactivity allows users to explore the data
+    and analysis results in detail.
+- **Sharing results** - Spyglass enables sharing of data and analysis results
+    via [Kachery](https://github.com/flatironinstitute/kachery-cloud), a
+    decentralized content addressable data sharing platform. Kachery Cloud
+    allows users to access the database and pull data and analysis results
+    directly to their local machine.
+- **Pipeline versioning** - Processing and analysis of data in neuroscience is
+    often dynamic, requiring new features. Spyglass uses *Merge tables* to
+    ensure that analysis pipelines can be versioned. This allows users to easily
+    use and compare results from different versions of the analysis pipeline
+    while retaining the ability to access previously generated results.
+- **Cautious Delete** - Spyglass uses a `cautious delete` feature to ensure that
+    data is not accidentally deleted by other users. When a user deletes data,
+    Spyglass will first check to see if the data belongs to another team of
+    users. This enables teams of users to work collaboratively on the same
+    database without worrying about accidentally deleting each other's data.
 
 ## Getting Started
 

docs/src/installation.md

@@ -25,7 +25,7 @@ pip install spikeinterface[full,widgets]
 pip install mountainsort4
 ```
 
-WARNING: If you are on an M1 Mac, you need to install `pyfftw` via `conda`
+__WARNING:__ If you are on an M1 Mac, you need to install `pyfftw` via `conda`
 BEFORE installing `ghostipy`:
 
 ```bash
@@ -49,40 +49,30 @@ additional details, see the
 
 #### Via File (Recommended)
 
-A `dj_local_conf.json` file in your Spyglass directory (or wherever python is
-launched) can hold all the specifics needed to connect to a database. This can
-include different directories for different pipelines. If only the `base` is
-specified, the subfolder names below are included as defaults.
-
-```json
-{
-  "custom": {
-    "database.prefix": "username_",
-    "spyglass_dirs": {
-      "base": "/your/base/path",
-      "raw": "/your/base/path/raw",
-      "analysis": "/your/base/path/analysis",
-      "recording": "/your/base/path/recording",
-      "spike_sorting_storage": "/your/base/path/spikesorting",
-      "waveforms": "/your/base/path/waveforms",
-      "temp": "/your/base/path/tmp"
-    }
-  }
-}
-```
-
-`dj_local_conf_example.json` can be copied and saved as `dj_local_conf.json` to
-set the configuration for a given folder. Alternatively, it can be saved as
-`.datajoint_config.json` in a user's home directory to be accessed globally. See
+A `dj_local_conf.json` file in your current directory when launching python can
+hold all the specifics needed to connect to a database. This can include
+different directories for different pipelines. If only the Spyglass `base` is
+specified, other subfolder names are assumed from defaults. See
+`dj_local_conf_example.json` for the full set of options. This example can be
+copied and saved as `dj_local_conf.json` to set the configuration for a given
+folder. Alternatively, it can be saved as `.datajoint_config.json` in a user's
+home directory to be accessed globally. See
 [DataJoint docs](https://datajoint.com/docs/core/datajoint-python/0.14/quick-start/#connection)
 for more details.
 
+Note that raw and analysis folder locations should be specified under both
+`stores` and `custom` sections of the config file. The `stores` section is used
+by DataJoint to store the location of files referenced in database, while the
+`custom` section is used by Spyglass. Spyglass will check that these sections
+match on startup.
+
 #### Via Environment Variables
 
 Older versions of Spyglass relied exclusively on environment for config. If
 `spyglass_dirs` is not found in the config file, Spyglass will look for
 environment variables. These can be set either once in a terminal session, or
-permanently in a `.bashrc` file.
+permanently in a unix settings file (e.g., `.bashrc` or `.bash_profile`) in your
+home directory.
 
 ```bash
 export SPYGLASS_BASE_DIR="/stelmo/nwb"
@@ -102,14 +92,21 @@ A temporary directory will speed up spike sorting. If unspecified by either
 method above, it will be assumed as a `tmp` subfolder relative to the base path.
 Be sure it has enough free space (ideally at least 500GB).
 
+#### Subfolders
+
+If subfolders do not exist, they will be created automatically. If unspecified
+by either method above, they will be assumed as `recording`, `sorting`, `video`,
+etc. subfolders relative to the base path.
+
 ## File manager
 
 [`kachery-cloud`](https://github.com/flatironinstitute/kachery-cloud) is a file
 manager for Frank Lab collaborators who do not have access to the lab's
 production database.
 
-To customize `kachery` file paths, the following can similarly be pasted into
-your `.bashrc`. If unspecified, the defaults below are assumed.
+To customize `kachery` file paths, see `dj_local_conf_example.json` or set the
+following variables in your unix settings file (e.g., `.bashrc`). If
+unspecified, the defaults below are assumed.
 
 ```bash
 export KACHERY_CLOUD_DIR="$SPYGLASS_BASE_DIR/.kachery-cloud"
@@ -122,3 +119,9 @@ Be sure to load these with `source ~/.bashrc` to persist changes.
 
 Finally, open up a python console (e.g., run `ipython` from terminal) and import
 `spyglass` to check that the installation has worked.
+
+```python
+from spyglass.common import Nwbfile
+
+Nwbfile()
+```