From 36769caf7302e8f35323bc0a4f22195de7d0b193 Mon Sep 17 00:00:00 2001 From: zm711 <92116279+zm711@users.noreply.github.com> Date: Sat, 4 Nov 2023 13:52:29 -0400 Subject: [PATCH 1/9] add import docs --- doc/import.rst | 81 ++++++++++++++++++++++++++++++++++++++++++++ doc/index.rst | 1 + doc/installation.rst | 12 ++++--- doc/modules/core.rst | 31 ----------------- 4 files changed, 90 insertions(+), 35 deletions(-) create mode 100644 doc/import.rst diff --git a/doc/import.rst b/doc/import.rst new file mode 100644 index 0000000000..8024ae5be8 --- /dev/null +++ b/doc/import.rst @@ -0,0 +1,81 @@ +Importing SpikeInterface +======================== + +SpikeInterface allows for the generation of powerful and reproducible spike sorting pipelines. +Flexibility is built into the package starting from import to maximize the productivity of +the developer and the scientist. Thus there are three ways that SpikeInterface and its components +can be imported: + + +Importing by Module +------------------- + +Since each spike sorting pipeline involves a series of often repeated steps, many of the developers +working on SpikeInterface recommend importing in a module by module fashion. This will allow you to +keep track of your processing steps (preprocessing, postprocessing, quality metrics, etc.). This can +be accomplished by: + +.. code-block:: python + + import spikeinterface as si + +to import the :code:`core` module followed by: + +.. code-block:: python + + import spikeinterface.extractors as se + import spikeinterface.preprocessing as spre + import spikeinterface.sorters as ss + import spikinterface.postprocessing as spost + import spikeinterface.qualitymetrics as sqm + import spikeinterface.exporters as sexp + import spikeinterface.comparsion as scmp + import spikeinterface.curation as scur + import spikeinterface.sortingcomponents as sc + import spikeinterface.widgets as sw + +to import any of the other modules you wish to use. + +The benefit of this approach is that it is lighter and faster than importing the whole package and allows +you to choose which of the modules you actually want to use. If you don't plan to export the results out of +SpikeInterface than you don't have to :code:`import spikeinterface.exporters`. Additionally the documentation +of the package is set-up in a modular fashion, so if you have a problem with :code:`spikeinterface.curation`, +you will know to go to the :code:`curation` section of this documention. The disadvantage of this approach is +that you have more aliases to keep track of. + + +Flat Import +----------- + +A second option is to import the SpikeInterface package in :code:`full` mode. This would be similar to +what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`). To accomplish this one does: + + +.. code-block:: python + + import spikeinterface.full as si + + +This import statement will import all of SpikeInterface modules as one flattened module. +Note that importing :code:`spikeinterface.full` will take a few extra seconds, because some modules use +just-in-time :code:`numba` compilation performed at the time of import. +We recommend this approach for advanced users, since it requires a deeper knowledge of the API. The advantage +being that users with advanced API knowledge can access all functions using one alias. + + +Importing Individual Functions +------------------------------ + +Finally, some users may find it useful to have extremely light imports and only import the exact functions +they plan to use. This can easily be accomplished by importing functions directly into the name space. + +For example: + +.. code-block:: python + + from spikeinterface.preprocessing import bandpass_filter, common_reference + from spikeinterface.core import extract_waveforms + from spikeinterface.extractors import read_binary + +As mentioned this approach only imports exactly what you plan on using so is the most minimalist. It does require +knowledge of the API to know which module to pull a function from. diff --git a/doc/index.rst b/doc/index.rst index df76a1a4c2..57ce14ed44 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -40,6 +40,7 @@ SpikeInterface is made of several modules to deal with different aspects of the overview installation + import modules/index how_to/index modules_gallery/index diff --git a/doc/installation.rst b/doc/installation.rst index acc5117249..05472591be 100644 --- a/doc/installation.rst +++ b/doc/installation.rst @@ -14,7 +14,7 @@ To install the current release version, you can use: The :code:`[full]` option installs all the extra dependencies for all the different sub-modules. -Note that if using Z shell (:code:`zsh` - the default shell on mac), you will need to use quotes (:code:`pip install "spikeinterface[full]"`). +Note that if using Z shell (:code:`zsh` - the default shell on macOS), you will need to use quotes (:code:`pip install "spikeinterface[full]"`). To install all interactive widget backends, you can use: @@ -63,14 +63,14 @@ as :code:`spikeinterface` strongly relies on these packages to interface with va It is also sometimes useful to have local copies of :code:`neo` and :code:`probeinterface` to make changes to the code. To achieve this, repeat the first set of commands, -replacing `https://github.com/SpikeInterface/spikeinterface.git` with the appropriate repository in the first code block of this section. +replacing :code:`https://github.com/SpikeInterface/spikeinterface.git` with the appropriate repository in the first code block of this section. For beginners ------------- We provide some installation tips for beginners in Python here: -https://github.com/SpikeInterface/spikeinterface/tree/master/installation_tips +https://github.com/SpikeInterface/spikeinterface/tree/main/installation_tips @@ -89,12 +89,16 @@ Requirements Sub-modules have more dependencies, so you should also install: * zarr + * h5py * scipy * pandas * xarray - * sklearn + * scikit-learn * networkx * matplotlib + * numba + * distinctipy + * cude-python (for non-macOS users) All external spike sorters can be either run inside containers (Docker or Singularity - see :ref:`containerizedsorters`) diff --git a/doc/modules/core.rst b/doc/modules/core.rst index 4c03950b1d..b25648a7a0 100644 --- a/doc/modules/core.rst +++ b/doc/modules/core.rst @@ -22,37 +22,6 @@ All classes support: * multiple segments, where each segment is a contiguous piece of data (recording, sorting, events). -Import rules ------------- - -Importing the SpikeInterface module - -.. code-block:: python - - import spikeinterface as si - -will only import the :code:`core` module. Other submodules must be imported separately: - -.. code-block:: python - - import spikeinterface.extractors as se - import spikeinterface.sorters as ss - import spikeinterface.widgets as sw - - -A second option is to import the SpikeInterface package in :code:`full` mode: - -.. code-block:: python - - import spikeinterface.full as si - -This import statement will import all of SpikeInterface modules as a flattened module. -Note that importing :code:`spikeinterface.full` will take a few extra seconds, because some modules use -just-in-time :code:`numba` compilation performed at the time of import. -We recommend this approach to advanced users, since it requires a deeper knowledge of the API. - - - Recording --------- From c40b3ad715ac2a57edf67b48a71cafd99a47603b Mon Sep 17 00:00:00 2001 From: Zach McKenzie <92116279+zm711@users.noreply.github.com> Date: Wed, 8 Nov 2023 08:02:50 -0500 Subject: [PATCH 2/9] one round of edits --- doc/import.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/import.rst b/doc/import.rst index 8024ae5be8..eaeec6002e 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -38,8 +38,8 @@ to import any of the other modules you wish to use. The benefit of this approach is that it is lighter and faster than importing the whole package and allows you to choose which of the modules you actually want to use. If you don't plan to export the results out of -SpikeInterface than you don't have to :code:`import spikeinterface.exporters`. Additionally the documentation -of the package is set-up in a modular fashion, so if you have a problem with :code:`spikeinterface.curation`, +SpikeInterface then you don't have to :code:`import spikeinterface.exporters`. Additionally the documentation +of the package is set-up in a modular fashion, so if you have a problem with the module :code:`spikeinterface.curation`, you will know to go to the :code:`curation` section of this documention. The disadvantage of this approach is that you have more aliases to keep track of. @@ -56,11 +56,11 @@ what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`). To ac import spikeinterface.full as si -This import statement will import all of SpikeInterface modules as one flattened module. +This import statement will import all of the SpikeInterface modules as one flattened module. Note that importing :code:`spikeinterface.full` will take a few extra seconds, because some modules use just-in-time :code:`numba` compilation performed at the time of import. We recommend this approach for advanced users, since it requires a deeper knowledge of the API. The advantage -being that users with advanced API knowledge can access all functions using one alias. +being that users can access all functions using one alias. Importing Individual Functions @@ -77,5 +77,5 @@ For example: from spikeinterface.core import extract_waveforms from spikeinterface.extractors import read_binary -As mentioned this approach only imports exactly what you plan on using so is the most minimalist. It does require -knowledge of the API to know which module to pull a function from. +As mentioned this approach only imports exactly what you plan on using so it is the most minimalist. It does require +knowledge of the API to know which module to pull a function from. It could also lead to naming clashes if pulling functions directly from other scientific libraries. From 5d318fa22194d7d58dcc7abe1864d86418783e22 Mon Sep 17 00:00:00 2001 From: Zach McKenzie <92116279+zm711@users.noreply.github.com> Date: Wed, 8 Nov 2023 08:13:54 -0500 Subject: [PATCH 3/9] Sam's hot takes Co-authored-by: Garcia Samuel --- doc/import.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/import.rst b/doc/import.rst index eaeec6002e..5da9389df0 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -59,8 +59,8 @@ what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`). To ac This import statement will import all of the SpikeInterface modules as one flattened module. Note that importing :code:`spikeinterface.full` will take a few extra seconds, because some modules use just-in-time :code:`numba` compilation performed at the time of import. -We recommend this approach for advanced users, since it requires a deeper knowledge of the API. The advantage -being that users can access all functions using one alias. +We recommend this approach for advanced (or lazy) users, since it requires a deeper knowledge of the API. The advantage +being that users can access all functions using one alias without the need of memorizing all aliases. Importing Individual Functions From 17c91a27eb4d2063848d45573b19fb1bf7534f5e Mon Sep 17 00:00:00 2001 From: Zach McKenzie <92116279+zm711@users.noreply.github.com> Date: Thu, 9 Nov 2023 05:58:26 -0500 Subject: [PATCH 4/9] Heberto's fix Co-authored-by: Heberto Mayorquin --- doc/installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/installation.rst b/doc/installation.rst index 05472591be..08f9077a1d 100644 --- a/doc/installation.rst +++ b/doc/installation.rst @@ -98,7 +98,7 @@ Sub-modules have more dependencies, so you should also install: * matplotlib * numba * distinctipy - * cude-python (for non-macOS users) + * cuda-python (for non-macOS users) All external spike sorters can be either run inside containers (Docker or Singularity - see :ref:`containerizedsorters`) From 6126888e49795d58a6284e7a5540dafbc56264ca Mon Sep 17 00:00:00 2001 From: Zach McKenzie <92116279+zm711@users.noreply.github.com> Date: Thu, 9 Nov 2023 06:55:35 -0500 Subject: [PATCH 5/9] some updates based on feedback --- doc/import.rst | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/doc/import.rst b/doc/import.rst index 5da9389df0..19fc74c3a8 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -36,19 +36,22 @@ to import the :code:`core` module followed by: to import any of the other modules you wish to use. -The benefit of this approach is that it is lighter and faster than importing the whole package and allows -you to choose which of the modules you actually want to use. If you don't plan to export the results out of -SpikeInterface then you don't have to :code:`import spikeinterface.exporters`. Additionally the documentation -of the package is set-up in a modular fashion, so if you have a problem with the module :code:`spikeinterface.curation`, -you will know to go to the :code:`curation` section of this documention. The disadvantage of this approach is -that you have more aliases to keep track of. +The benefit of this approach is that it is lighter than importing the whole library as a flat module and allows +you to choose which of the modules you actually want to use. It also reminds you what step of the pipeline each +submodule is meant to be used for. If you don't plan to export the results out of SpikeInterface then you +don't have to :code:`import spikeinterface.exporters`. Additionally the documentation of SpikeInterface is set-up +in a modular fashion, so if you have a problem with the submodule :code:`spikeinterface.curation`,you will know +to go to the :code:`curation` section of this documention. The disadvantage of this approach is that you have +more aliases to keep track of. Flat Import ----------- A second option is to import the SpikeInterface package in :code:`full` mode. This would be similar to -what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`). To accomplish this one does: +what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`), which offer the majority of +their functionality with a single alias and the option to import additional functionality separately. +To accomplish this one does: .. code-block:: python @@ -78,4 +81,5 @@ For example: from spikeinterface.extractors import read_binary As mentioned this approach only imports exactly what you plan on using so it is the most minimalist. It does require -knowledge of the API to know which module to pull a function from. It could also lead to naming clashes if pulling functions directly from other scientific libraries. +knowledge of the API to know which module to pull a function from. It could also lead to naming clashes if pulling +functions directly from other scientific libraries. Type :code:`import this` for more information. From ec09322d4738557e104aee264988c1a477a212d2 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Thu, 9 Nov 2023 11:55:53 +0000 Subject: [PATCH 6/9] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/import.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/import.rst b/doc/import.rst index 19fc74c3a8..430e53ad4e 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -38,10 +38,10 @@ to import any of the other modules you wish to use. The benefit of this approach is that it is lighter than importing the whole library as a flat module and allows you to choose which of the modules you actually want to use. It also reminds you what step of the pipeline each -submodule is meant to be used for. If you don't plan to export the results out of SpikeInterface then you -don't have to :code:`import spikeinterface.exporters`. Additionally the documentation of SpikeInterface is set-up -in a modular fashion, so if you have a problem with the submodule :code:`spikeinterface.curation`,you will know -to go to the :code:`curation` section of this documention. The disadvantage of this approach is that you have +submodule is meant to be used for. If you don't plan to export the results out of SpikeInterface then you +don't have to :code:`import spikeinterface.exporters`. Additionally the documentation of SpikeInterface is set-up +in a modular fashion, so if you have a problem with the submodule :code:`spikeinterface.curation`,you will know +to go to the :code:`curation` section of this documention. The disadvantage of this approach is that you have more aliases to keep track of. @@ -49,8 +49,8 @@ Flat Import ----------- A second option is to import the SpikeInterface package in :code:`full` mode. This would be similar to -what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`), which offer the majority of -their functionality with a single alias and the option to import additional functionality separately. +what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`), which offer the majority of +their functionality with a single alias and the option to import additional functionality separately. To accomplish this one does: @@ -81,5 +81,5 @@ For example: from spikeinterface.extractors import read_binary As mentioned this approach only imports exactly what you plan on using so it is the most minimalist. It does require -knowledge of the API to know which module to pull a function from. It could also lead to naming clashes if pulling +knowledge of the API to know which module to pull a function from. It could also lead to naming clashes if pulling functions directly from other scientific libraries. Type :code:`import this` for more information. From 69f6f8ccebe2085703c918a05e4693778a391067 Mon Sep 17 00:00:00 2001 From: Zach McKenzie <92116279+zm711@users.noreply.github.com> Date: Fri, 17 Nov 2023 05:35:57 -0500 Subject: [PATCH 7/9] Sam & Heberto feedback --- doc/import.rst | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/doc/import.rst b/doc/import.rst index 430e53ad4e..fbf5c6f985 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -48,9 +48,7 @@ more aliases to keep track of. Flat Import ----------- -A second option is to import the SpikeInterface package in :code:`full` mode. This would be similar to -what is seen with packages like NumPy (:code:`np`) or Pandas (:code:`pd`), which offer the majority of -their functionality with a single alias and the option to import additional functionality separately. +A second option is to import the SpikeInterface package in :code:`full` mode. To accomplish this one does: @@ -60,8 +58,6 @@ To accomplish this one does: This import statement will import all of the SpikeInterface modules as one flattened module. -Note that importing :code:`spikeinterface.full` will take a few extra seconds, because some modules use -just-in-time :code:`numba` compilation performed at the time of import. We recommend this approach for advanced (or lazy) users, since it requires a deeper knowledge of the API. The advantage being that users can access all functions using one alias without the need of memorizing all aliases. From 7595d9ce3ba57f11ab33e8260f26654c1fca204a Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 17 Nov 2023 10:36:53 +0000 Subject: [PATCH 8/9] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- doc/import.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/import.rst b/doc/import.rst index fbf5c6f985..ad5e1da0ee 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -48,7 +48,7 @@ more aliases to keep track of. Flat Import ----------- -A second option is to import the SpikeInterface package in :code:`full` mode. +A second option is to import the SpikeInterface package in :code:`full` mode. To accomplish this one does: From 836d77b256b17aec7dae1009f341dc4e2b279f44 Mon Sep 17 00:00:00 2001 From: Zach McKenzie <92116279+zm711@users.noreply.github.com> Date: Fri, 17 Nov 2023 06:53:00 -0500 Subject: [PATCH 9/9] import spikeinterface.core as si --- doc/import.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/import.rst b/doc/import.rst index ad5e1da0ee..be3f7d5afb 100644 --- a/doc/import.rst +++ b/doc/import.rst @@ -17,7 +17,7 @@ be accomplished by: .. code-block:: python - import spikeinterface as si + import spikeinterface.core as si to import the :code:`core` module followed by: