Spike sorting v1 pipeline

[khl02007]

Add new sub-package called v1 within spikesorting that contains the following modules:

• recording.py
• artifact.py
• sorting.py
• curation.py
• metric_curation.py
• figurl_curation.py
  Also add the following modules in the spikesorting package:
• imported.py
• merge.py

General

• All the tables live in new schema with naming convention spikesorting_v1_modulename
• Make the name of tables singular (~Parameters $\to$ ~Parameter)
• Remove environment variables for saving preprocessed recordings, sortings, and waveforms; these are now saved in the same directory as other analysis NWB files
• Reorder steps in make methods to have the following structure:
  • Fetch (fetch info from database) - Do (process fetched info) - Insert (insert info back to database)
• Make most methods classmethod
• Add neuroconv to installation requirement (needed to write recording to NWB)

Module-specific

recording.py

SortGroup

• Move Session foreign key from primary to secondary key
• Remove methods
  • SortGroup.set_group_by_electrode_group
  • SortGroup.set_reference_from_list
  • SortGroup.get_geometry

SortInterval

• Remove; use IntervalList instead

SpikeSortingPreprocessingParameter

• Change parent class to dj.Lookup
• Add default parameters to contents property of the table

SpikeSortingRecording

• Use recording_id as the sole primary key; move other attributes that used to be primary key to secondary
• Write recording to NWB
• Add SpikeSortingRecording.get_recording method

artifact.py

ArtifactDetectionParameter

• Change parent class to dj.Lookup
• Add default parameters to contents property

ArtifactRemovedIntervalList

• Removed

ArtifactInterval

• Rename to ArtifactRemovedInterval
• Add new primary key: artifact_id
• No longer computes artifact intervals; only computes intervals free of artifacts
• No longer saves artifact free intervals; these are saved in IntervalList and interval_list_name is same as artifact_id

sorting.py

SpikeSorterParameter

• Change parent class to dj.Lookup

SpikeSortingSelection

• Inherit from IntervalList for artifact-free sorting interval

SpikeSorting

• Save sorting to an analysis NWB file and remove SPYGLASS_SORTING_DIR
• Remove clean up methods for SPYGLASS_SORTING_DIR

curation.py

Curation

• Change the secondary key to include parent_curation_id, merges_applied, and foreign key from AnalysisNwbfile
• dj.Manual table but using insert_curation method is recommended to populate the table; this function saves the sorting and curation information in NWB
• Four methods exposed to the user:
  • insert_curation
  • get_recording
  • get_sorting
  • get_merged_sorting: differs from get_sorting in that it applies merge groups to the sorting

metric_curation.py

• Move the automatic curation pipleline that used to be in spikesorting_curation.py to here
• Merge the waveform extraction, metric computation, and curation labels / merge group generation to a single compute table

MetricCuration

• Save the waveforms, quality metrics, curation labels and merge groups to NWB
• Has methods for accessing these from NWB

figurl_curation.py

• Creates new curation_uri.json each time. This contains the curation labels and merge groups to be shown in the figURL.

FigURLCurationSelection

• Contains metrics_figurl as primary key; specifies which metrics to include in the figURL
• Has generate_curation_uri method, which can be used to prepopulate the curation labels or merge groups from previous curation in the new figURL

FigURLCuration

• Only secondary key is url

To do

• Provide a way to access waveforms as spikeinterface.WaveformExtractor from MetricCuration table
• Provide a way to include similarity matrices to the figURL
• Write tutorial notebooks
• pynwb.units requires the unit ID to be int; sortingview requires unit IDs to be str; spikeinterface allows them to be either type. Write tests to make sure these conventions are respected.

[khl02007]

Thanks @CBroz1 @edeno @samuelbray32 for the comments. I have incorporated as many suggested changes as I could, except for some that threw errors (I reverted those back) and those that were outdated and could not be applied (e.g. the insert sort group by shank function). But otherwise I think this is ready to be merged. The code can obviously be optimized further but I say let's go for it and worry about that later!