From 1ba5b867545a45bf2690ba9dda5b19ae4589cf84 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Thu, 7 Dec 2023 23:48:59 +0000 Subject: [PATCH 1/8] Split by recording first commit. --- doc/how_to/index.rst | 1 + doc/how_to/process_by_channel_group.rst | 128 ++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 doc/how_to/process_by_channel_group.rst diff --git a/doc/how_to/index.rst b/doc/how_to/index.rst index da94cf549c..8c3a4fbb35 100644 --- a/doc/how_to/index.rst +++ b/doc/how_to/index.rst @@ -8,3 +8,4 @@ How to guides analyse_neuropixels handle_drift load_matlab_data + process_by_channel_group diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst new file mode 100644 index 0000000000..1e186d4e6b --- /dev/null +++ b/doc/how_to/process_by_channel_group.rst @@ -0,0 +1,128 @@ +Processing a Recording by Channel Group (e.g. Shank) +=========================================================== + +In this tutorial, we will walk through how to preprocess and sorting a recording +per channel group. A channel group is a subset of channels grouped by some +feature - a typical example is grouping channels by shank in a Neuropixels recording. + +First, we can check the channels on our recording are grouped as we expect. For example, +in the below example we have a X-shank neuropixels recording. We can visualise the +channel groupings with XXX. + + +On our SpikeInterface recording, we can also access the channel groups per-channel + +1) channel groups +2) index channel ids by groups + +Why would you want to preprocess or sort by group? + +1) +2) +3) + +Splitting a Recording by Channel Group +-------------------------------------- + +We can split a recording into mutliply recordings, one for each channel group, with the `split_by` method. + +Here, we split a recording by channel group to get a `split_recording_dict`. This is a dictionary +containing the recordings, split by channel group: + +``` +split_recording_dict = recording.split_by("group") +print(split_recording_dict) +XXXXX +``` + +Now, we are ready to preprocess and sort by channel group. + +Preprocessing a Recording by Channel Group +------------------------------------------ + +The essense of preprocessing by channel group is to first split the recording +into separate recordings, perform the preprocessing steps, then aggregate +the channels back together. + +Here, we loop over the split recordings, preprocessing each shank group +individually. At the end, we use the `aggregate_channels` function +to combine the per-shank recordings back together. + +``` +preprocessed_recordings = [] +for recording in split_recordings_dict.values(): + + + shifted_recording = spre.phase_shift(recording) + + filtered_recording = spre.bandpass_filter(shifted_recording) + + referenced_recording = spre.common_reference(filtered_recording) + + preprocessed_recordings.append(referenced_recording) + +combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) + +``` + +It is strongly recommended to use the above structure to preprocess by group. +A discussion of the subtleties of the approach may be found in the below +Notes section for the interested reader. + +# Preprocessing channel in depth + +Preprocessing and aggregation of channels is very flexible. Under the hood, +`aggregate_channels` keeps track of when a recording was split. When `get_traces()` +is called, the preprocessing is still performed per-group, even though the +recording is now aggregated. + +However, to ensure data is preprocessed by shank, the preprocessing step must be +applied per-group. For example, the below example will NOT preprocess by shank: + +``` +split_recording = recording.split_by("group") +split_recording_as_list = list(**split_recording.values()) +combined_recording = aggregate_channels(split_recording_as_list) + +# will NOT preprocess by channel group. +filtered_recording = common_reference(combined_recording) + +``` + +Similarly, in the below example the first preprocessing step (bandpass filter) +would applied by group (although, this would have no effect in practice +as this preprocessing step is always performed per channel). However, +common referencing (which is effected by splitting by group) will +not be applied per group: + +``` +split_recording = recording.split_by("group") + +filtered_recording = [] +for recording in split_recording.values() + filtered_recording.append(spre.bandpass_filtered(recording)) + +combined_recording = aggregate_channels(filtered_recording) + +# As the recording has been combined, common referencing +# will NOT be applied per channel group. +referenced_recording = spre.common_reference(combined_recording). +``` + +Finally, it is not recommended to apply `aggregate_channels` more than once +as this will slow down `get_traces()` and may result in unpredictable behaviour. + + +Sorting a Recording by Channel Group +------------------------------------ + +Sorting a recording can be performed in two ways. One, is to split the +recording by group and use `run_sorter` (LINK) separately on each preprocessed +channel group. + +Altearntively, SpikeInterface proves a conveniecne function XXX +to this. + +Example + +Done! From 9a14de55af43301760b3e1854c4aaca6f55ffc26 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Fri, 8 Dec 2023 12:05:27 +0000 Subject: [PATCH 2/8] Reword and tidy up, currently fixing recording issue. --- doc/how_to/process_by_channel_group.rst | 239 ++++++++++++++++-------- doc/sg_execution_times.rst | 82 ++++++++ 2 files changed, 240 insertions(+), 81 deletions(-) create mode 100644 doc/sg_execution_times.rst diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index 1e186d4e6b..8e33afa207 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -1,128 +1,205 @@ -Processing a Recording by Channel Group (e.g. Shank) -=========================================================== +Processing a Recording by Channel Group +======================================= -In this tutorial, we will walk through how to preprocess and sorting a recording -per channel group. A channel group is a subset of channels grouped by some -feature - a typical example is grouping channels by shank in a Neuropixels recording. +In this tutorial, we will walk through how to preprocess and sort a recording +separately for *channel groups*. A channel group is a subset of channels grouped by some +feature - for example a multi-shank Neuropixels recording in which the channels +are grouped by shank. -First, we can check the channels on our recording are grouped as we expect. For example, -in the below example we have a X-shank neuropixels recording. We can visualise the -channel groupings with XXX. +**Why preprocess by channel group?** +Certain preprocessing steps depend on the spatial arrangement of the channels. +For example, common average referencing (CAR) averages over channels (separately for each time point) +and subtracts the average. In such a scenario it may make sense to group channels so that +this averaging is performed only over spatially close channel groups. -On our SpikeInterface recording, we can also access the channel groups per-channel +**Why sort by channel group?** -1) channel groups -2) index channel ids by groups +When sorting, we may want to completely separately channel groups so we can +consider their signals in isolation. If recording from a long +silicon probe, we might want to sort different brain areas separately, +for example using a different sorter for the hippocampus, the thalamus, or the cerebellum. -Why would you want to preprocess or sort by group? - -1) -2) -3) Splitting a Recording by Channel Group -------------------------------------- -We can split a recording into mutliply recordings, one for each channel group, with the `split_by` method. +In this example, we create a 16-channel recording with 4 tetrodes. However this could +be any recording in which the channel are grouped in some way, for example +a 384 channel, 4 shank Neuropixels recording in which channel grouping represents the separate shanks. + +First, let's import the parts of SpikeInterface we need into Python, and generate our toy recording: + +.. code-block:: python + + import spikeinterface.extractors as se + import spikeinterface.preprocessing as spre + from spikeinterface import aggregate_channels + from probeinterface import generate_tetrode, ProbeGroup + import numpy as np + + recording, _ = se.toy_example(duration=[10.], num_segments=1, num_channels=16) + print(recording.get_channel_groups()) + # >>> [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] (here, all channels are in the same group) + + # create 4 tetrodes as a 'probegroup' + probegroup = ProbeGroup() + for i in range(4): + tetrode = generate_tetrode() + tetrode.set_device_channel_indices(np.arange(4) + i * 4) + probegroup.add_probe(tetrode) + + # set this to the recording + recording_4_tetrodes = recording.set_probegroup(probegroup, group_mode='by_probe') -Here, we split a recording by channel group to get a `split_recording_dict`. This is a dictionary -containing the recordings, split by channel group: + print(recording_4_tetrodes.get_property('group')) + # >>> [0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3] (now, channels are grouped by tetrode index) -``` -split_recording_dict = recording.split_by("group") -print(split_recording_dict) -XXXXX -``` +We can split a recording into multiple recordings, one for each channel group, with the :py:func:`~split_by` method. -Now, we are ready to preprocess and sort by channel group. +.. code-block:: python + split_recording_dict = recording.split_by("group") + +Splitting a recording by channel group returns a dictionary containing separate recordings, one for each channel group: + +.. code-block:: python + + print(split_recording_dict) + # {0: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB, 1: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB, 2: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB, 3: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB} Preprocessing a Recording by Channel Group ------------------------------------------ -The essense of preprocessing by channel group is to first split the recording +The essence of preprocessing by channel group is to first split the recording into separate recordings, perform the preprocessing steps, then aggregate the channels back together. -Here, we loop over the split recordings, preprocessing each shank group -individually. At the end, we use the `aggregate_channels` function -to combine the per-shank recordings back together. +In the below example, we loop over the split recordings, preprocessing each channel group +individually. At the end, we use the :py:func:`~aggregate_channels` function +to combine the separate channel group recordings back together. -``` -preprocessed_recordings = [] -for recording in split_recordings_dict.values(): +.. code-block:: python + preprocessed_recordings = [] - shifted_recording = spre.phase_shift(recording) + # loop over the recordings contained in the dictionary + for chan_group_rec in split_recordings_dict.values(): - filtered_recording = spre.bandpass_filter(shifted_recording) + # Apply the preprocessing steps to the channel group in isolation + shifted_recording = spre.phase_shift(chan_group_rec) - referenced_recording = spre.common_reference(filtered_recording) + filtered_recording = spre.bandpass_filter(shifted_recording) - preprocessed_recordings.append(referenced_recording) + referenced_recording = spre.common_reference(filtered_recording) -combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) + preprocessed_recordings.append(referenced_recording) -``` + # Combine our preprocessed channel groups back together + combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) -It is strongly recommended to use the above structure to preprocess by group. -A discussion of the subtleties of the approach may be found in the below -Notes section for the interested reader. +Now, when this recording is used in sorting, plotting, or whenever +calling it's :py:func:`~get_traces` method, the data will have been +preprocessed separately per-channel group (then concatenated +back together under the hood). -# Preprocessing channel in depth +It is strongly recommended to use the above structure to preprocess by channel group. +A discussion of the subtleties of the :py:func:`~aggregate_channels` may be found in +the below section for the interested reader. -Preprocessing and aggregation of channels is very flexible. Under the hood, -`aggregate_channels` keeps track of when a recording was split. When `get_traces()` -is called, the preprocessing is still performed per-group, even though the -recording is now aggregated. +.. warning:: + It is not recommended to apply :py:func:`~aggregate_channels` more than once + as this will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. -However, to ensure data is preprocessed by shank, the preprocessing step must be -applied per-group. For example, the below example will NOT preprocess by shank: -``` -split_recording = recording.split_by("group") -split_recording_as_list = list(**split_recording.values()) -combined_recording = aggregate_channels(split_recording_as_list) +Sorting a Recording by Channel Group +------------------------------------ -# will NOT preprocess by channel group. -filtered_recording = common_reference(combined_recording) +We can also sort a recording for each channel group separately. It is not necessary to preprocess +a recording by channel group in order to sort by channel group. We can perform sorting on a recording +whichever way it was preprocessed. -``` +There are two ways to sort a recording by channel group. First, we can split the preprocessed +recording (or, if it was already split during preprocessing as above, skip the :py:func:`~aggregate_channels` step +directly use the :py:func:`~split_recording_dict`). -Similarly, in the below example the first preprocessing step (bandpass filter) -would applied by group (although, this would have no effect in practice -as this preprocessing step is always performed per channel). However, -common referencing (which is effected by splitting by group) will -not be applied per group: +**Option 1: Manual splitting** -``` -split_recording = recording.split_by("group") +In this example, similar to above we loop over all preprocessed recordings that +are groups by channel, and apply the sorting separately. We store the +sorting objects in a dictionary for later use. -filtered_recording = [] -for recording in split_recording.values() - filtered_recording.append(spre.bandpass_filtered(recording)) +.. code-block:: python -combined_recording = aggregate_channels(filtered_recording) + split_preprocessed_recording = preprocessed_recording.split_by("group") -# As the recording has been combined, common referencing -# will NOT be applied per channel group. -referenced_recording = spre.common_reference(combined_recording). -``` + sortings = {} + for group, sub_recording in split_preprocessed_recording.items(): + sorting = run_sorter( + sorter_name='kilosort2', + recording=split_preprocessed_recording, + output_folder=f"folder_KS2_group{group}" + ) + sortings[group] = sorting -Finally, it is not recommended to apply `aggregate_channels` more than once -as this will slow down `get_traces()` and may result in unpredictable behaviour. +**Option 2 : Automatic splitting** +Alternatively, SpikeInterface provides a convenience function to sort the recording by property: -Sorting a Recording by Channel Group ------------------------------------- +.. code-block:: python + + aggregate_sorting = run_sorter_by_property( + sorter_name='kilosort2', + recording=recording_4_tetrodes, + grouping_property='group', + working_folder='working_path' + ) + + +Further notes on preprocessing by channel group +----------------------------------------------- + +.. note:: + + The splitting and aggregation of channels for preprocessing is flexible. + Under the hood, :py:func:`~aggregate_channels` keeps track of when a recording was split. When + :py:func:`~get_traces` is called, the preprocessing is still performed per-group, + even though the recording is now aggregated. + + However, to ensure data is preprocessed by channel group, the preprocessing step must be + applied separately to each split channel group recording. + For example, the below example will NOT preprocess by shank: + + .. code-block:: python + + split_recording = recording.split_by("group") + split_recording_as_list = list(**split_recording.values()) + combined_recording = aggregate_channels(split_recording_as_list) + + # will NOT preprocess by channel group. + filtered_recording = common_reference(combined_recording) + + Similarly, in the below example the first preprocessing step (bandpass filter) + would applied by group (although, this would have no effect in practice + as this preprocessing step is always separately for each individual channel). + + However, in the below example common referencing (does operate across + separate channels) will not be applied per channel group: + + .. code-block:: python -Sorting a recording can be performed in two ways. One, is to split the -recording by group and use `run_sorter` (LINK) separately on each preprocessed -channel group. + split_recording = recording.split_by("group") -Altearntively, SpikeInterface proves a conveniecne function XXX -to this. + filtered_recording = [] + for recording in split_recording.values() + filtered_recording.append(spre.bandpass_filtered(recording)) -Example + combined_recording = aggregate_channels(filtered_recording) -Done! + # As the recording has been combined, common referencing + # will NOT be applied per channel group. But, the bandpass + # filter step will be applied per channel group. + referenced_recording = spre.common_reference(combined_recording). diff --git a/doc/sg_execution_times.rst b/doc/sg_execution_times.rst new file mode 100644 index 0000000000..ac9bb37470 --- /dev/null +++ b/doc/sg_execution_times.rst @@ -0,0 +1,82 @@ + +:orphan: + +.. _sphx_glr_sg_execution_times: + + +Computation times +================= +**00:04.653** total execution time for 16 files **from all galleries**: + +.. container:: + + .. raw:: html + + + + + + + + .. list-table:: + :header-rows: 1 + :class: table table-striped sg-datatable + + * - Example + - Time + - Mem (MB) + * - :ref:`sphx_glr_modules_gallery_core_plot_4_waveform_extractor.py` (``..\examples\modules_gallery\core\plot_4_waveform_extractor.py``) + - 00:02.203 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) + - 00:00.856 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_extractors_plot_1_read_various_formats.py` (``..\examples\modules_gallery\extractors\plot_1_read_various_formats.py``) + - 00:00.675 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_3_quality_mertics.py` (``..\examples\modules_gallery\qualitymetrics\plot_3_quality_mertics.py``) + - 00:00.337 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_3_waveforms_gallery.py` (``..\examples\modules_gallery\widgets\plot_3_waveforms_gallery.py``) + - 00:00.335 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) + - 00:00.247 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_comparison_generate_erroneous_sorting.py` (``..\examples\modules_gallery\comparison\generate_erroneous_sorting.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_comparison_plot_5_comparison_sorter_weaknesses.py` (``..\examples\modules_gallery\comparison\plot_5_comparison_sorter_weaknesses.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_1_recording_extractor.py` (``..\examples\modules_gallery\core\plot_1_recording_extractor.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_2_sorting_extractor.py` (``..\examples\modules_gallery\core\plot_2_sorting_extractor.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_3_handle_probe_info.py` (``..\examples\modules_gallery\core\plot_3_handle_probe_info.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_5_append_concatenate_segments.py` (``..\examples\modules_gallery\core\plot_5_append_concatenate_segments.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_6_handle_times.py` (``..\examples\modules_gallery\core\plot_6_handle_times.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_extractors_plot_2_working_with_unscaled_traces.py` (``..\examples\modules_gallery\extractors\plot_2_working_with_unscaled_traces.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_1_rec_gallery.py` (``..\examples\modules_gallery\widgets\plot_1_rec_gallery.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_2_sort_gallery.py` (``..\examples\modules_gallery\widgets\plot_2_sort_gallery.py``) + - 00:00.000 + - 0.0 From b4faa9af698617ba0e1cea90e13f1372699828e5 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Mon, 11 Dec 2023 13:56:13 +0000 Subject: [PATCH 3/8] Update example to be neuropixels-like. --- doc/how_to/process_by_channel_group.rst | 42 ++++++++++++++++--------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index 8e33afa207..4530b4129b 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -38,22 +38,34 @@ First, let's import the parts of SpikeInterface we need into Python, and generat from probeinterface import generate_tetrode, ProbeGroup import numpy as np - recording, _ = se.toy_example(duration=[10.], num_segments=1, num_channels=16) - print(recording.get_channel_groups()) - # >>> [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] (here, all channels are in the same group) - - # create 4 tetrodes as a 'probegroup' - probegroup = ProbeGroup() - for i in range(4): - tetrode = generate_tetrode() - tetrode.set_device_channel_indices(np.arange(4) + i * 4) - probegroup.add_probe(tetrode) + # Create a toy 384 channel recording with 4 shanks (each shank contain 96 channels) + recording, _ = se.toy_example(duration=[1.00], num_segments=1, num_channels=384) + four_shank_groupings = np.repeat([0, 1, 2, 3], 96) + recording.set_property("group", four_shank_groupings) - # set this to the recording - recording_4_tetrodes = recording.set_probegroup(probegroup, group_mode='by_probe') + recording.get_property("location") - print(recording_4_tetrodes.get_property('group')) - # >>> [0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3] (now, channels are grouped by tetrode index) + print(recording.get_channel_groups()) + """ + array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]) + """ We can split a recording into multiple recordings, one for each channel group, with the :py:func:`~split_by` method. @@ -153,7 +165,7 @@ Alternatively, SpikeInterface provides a convenience function to sort the record aggregate_sorting = run_sorter_by_property( sorter_name='kilosort2', - recording=recording_4_tetrodes, + recording=preprocessed_recording, grouping_property='group', working_folder='working_path' ) From 1506fdad464a68e323d1849f1461439f3276bc11 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Mon, 11 Dec 2023 14:05:35 +0000 Subject: [PATCH 4/8] Tidy up and check. --- doc/how_to/process_by_channel_group.rst | 41 ++++++++++++------------- doc/sg_execution_times.rst | 20 ++++++------ 2 files changed, 30 insertions(+), 31 deletions(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index 4530b4129b..f3a8db615c 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -24,9 +24,9 @@ for example using a different sorter for the hippocampus, the thalamus, or the c Splitting a Recording by Channel Group -------------------------------------- -In this example, we create a 16-channel recording with 4 tetrodes. However this could +In this example, we create a 384-channel recording with 4 shanks. However this could be any recording in which the channel are grouped in some way, for example -a 384 channel, 4 shank Neuropixels recording in which channel grouping represents the separate shanks. +a multi-tetrode recording with channel groups representing the channels on each individual tetrodes. First, let's import the parts of SpikeInterface we need into Python, and generate our toy recording: @@ -43,8 +43,6 @@ First, let's import the parts of SpikeInterface we need into Python, and generat four_shank_groupings = np.repeat([0, 1, 2, 3], 96) recording.set_property("group", four_shank_groupings) - recording.get_property("location") - print(recording.get_channel_groups()) """ array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, @@ -78,11 +76,13 @@ Splitting a recording by channel group returns a dictionary containing separate .. code-block:: python print(split_recording_dict) - # {0: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB, 1: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB, 2: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB, 3: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB} + """ + {0: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB, 1: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB, 2: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB, 3: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB} + """ Preprocessing a Recording by Channel Group ------------------------------------------ @@ -120,19 +120,18 @@ back together under the hood). It is strongly recommended to use the above structure to preprocess by channel group. A discussion of the subtleties of the :py:func:`~aggregate_channels` may be found in -the below section for the interested reader. +the end of this tutorial for the interested reader. .. warning:: - It is not recommended to apply :py:func:`~aggregate_channels` more than once - as this will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. + It is not recommended to apply :py:func:`~aggregate_channels` more than once. + This will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. Sorting a Recording by Channel Group ------------------------------------ We can also sort a recording for each channel group separately. It is not necessary to preprocess -a recording by channel group in order to sort by channel group. We can perform sorting on a recording -whichever way it was preprocessed. +a recording by channel group in order to sort by channel group. There are two ways to sort a recording by channel group. First, we can split the preprocessed recording (or, if it was already split during preprocessing as above, skip the :py:func:`~aggregate_channels` step @@ -141,7 +140,7 @@ directly use the :py:func:`~split_recording_dict`). **Option 1: Manual splitting** In this example, similar to above we loop over all preprocessed recordings that -are groups by channel, and apply the sorting separately. We store the +are grouped by channel, and apply the sorting separately. We store the sorting objects in a dictionary for later use. .. code-block:: python @@ -181,9 +180,9 @@ Further notes on preprocessing by channel group :py:func:`~get_traces` is called, the preprocessing is still performed per-group, even though the recording is now aggregated. - However, to ensure data is preprocessed by channel group, the preprocessing step must be + To ensure data is preprocessed by channel group, the preprocessing step must be applied separately to each split channel group recording. - For example, the below example will NOT preprocess by shank: + For example, the below example will NOT preprocess by channel group: .. code-block:: python @@ -194,11 +193,11 @@ Further notes on preprocessing by channel group # will NOT preprocess by channel group. filtered_recording = common_reference(combined_recording) - Similarly, in the below example the first preprocessing step (bandpass filter) - would applied by group (although, this would have no effect in practice - as this preprocessing step is always separately for each individual channel). + In the below example the first preprocessing step (bandpass filter) + would applied by channel group (although, in practice this would have no effect + as filtering is always applied separately to each individual channel). - However, in the below example common referencing (does operate across + However, common referencing (does operate across separate channels) will not be applied per channel group: .. code-block:: python diff --git a/doc/sg_execution_times.rst b/doc/sg_execution_times.rst index ac9bb37470..38453ec0e7 100644 --- a/doc/sg_execution_times.rst +++ b/doc/sg_execution_times.rst @@ -6,7 +6,7 @@ Computation times ================= -**00:04.653** total execution time for 16 files **from all galleries**: +**00:03.838** total execution time for 16 files **from all galleries**: .. container:: @@ -33,22 +33,22 @@ Computation times - Time - Mem (MB) * - :ref:`sphx_glr_modules_gallery_core_plot_4_waveform_extractor.py` (``..\examples\modules_gallery\core\plot_4_waveform_extractor.py``) - - 00:02.203 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) - - 00:00.856 + - 00:02.038 - 0.0 * - :ref:`sphx_glr_modules_gallery_extractors_plot_1_read_various_formats.py` (``..\examples\modules_gallery\extractors\plot_1_read_various_formats.py``) - - 00:00.675 + - 00:00.640 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) + - 00:00.303 - 0.0 * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_3_quality_mertics.py` (``..\examples\modules_gallery\qualitymetrics\plot_3_quality_mertics.py``) - - 00:00.337 + - 00:00.301 - 0.0 * - :ref:`sphx_glr_modules_gallery_widgets_plot_3_waveforms_gallery.py` (``..\examples\modules_gallery\widgets\plot_3_waveforms_gallery.py``) - - 00:00.335 + - 00:00.285 - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) - - 00:00.247 + * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) + - 00:00.271 - 0.0 * - :ref:`sphx_glr_modules_gallery_comparison_generate_erroneous_sorting.py` (``..\examples\modules_gallery\comparison\generate_erroneous_sorting.py``) - 00:00.000 From 47049c92e4e38e84f18443a72374d414802598cf Mon Sep 17 00:00:00 2001 From: Alessio Buccino Date: Fri, 19 Jan 2024 11:17:34 +0100 Subject: [PATCH 5/8] Update doc/how_to/process_by_channel_group.rst Co-authored-by: Heberto Mayorquin --- doc/how_to/process_by_channel_group.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index f3a8db615c..c5c60ab889 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -194,7 +194,7 @@ Further notes on preprocessing by channel group filtered_recording = common_reference(combined_recording) In the below example the first preprocessing step (bandpass filter) - would applied by channel group (although, in practice this would have no effect + would be applied by channel group (although, in practice this would have no effect as filtering is always applied separately to each individual channel). However, common referencing (does operate across From 9feee09067b7954225ec0c48bbbb8fd75475a9bc Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Tue, 13 Feb 2024 11:47:43 +0000 Subject: [PATCH 6/8] Remove execution times .rst --- doc/sg_execution_times.rst | 82 -------------------------------------- 1 file changed, 82 deletions(-) delete mode 100644 doc/sg_execution_times.rst diff --git a/doc/sg_execution_times.rst b/doc/sg_execution_times.rst deleted file mode 100644 index 38453ec0e7..0000000000 --- a/doc/sg_execution_times.rst +++ /dev/null @@ -1,82 +0,0 @@ - -:orphan: - -.. _sphx_glr_sg_execution_times: - - -Computation times -================= -**00:03.838** total execution time for 16 files **from all galleries**: - -.. container:: - - .. raw:: html - - - - - - - - .. list-table:: - :header-rows: 1 - :class: table table-striped sg-datatable - - * - Example - - Time - - Mem (MB) - * - :ref:`sphx_glr_modules_gallery_core_plot_4_waveform_extractor.py` (``..\examples\modules_gallery\core\plot_4_waveform_extractor.py``) - - 00:02.038 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_extractors_plot_1_read_various_formats.py` (``..\examples\modules_gallery\extractors\plot_1_read_various_formats.py``) - - 00:00.640 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) - - 00:00.303 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_3_quality_mertics.py` (``..\examples\modules_gallery\qualitymetrics\plot_3_quality_mertics.py``) - - 00:00.301 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_3_waveforms_gallery.py` (``..\examples\modules_gallery\widgets\plot_3_waveforms_gallery.py``) - - 00:00.285 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) - - 00:00.271 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_comparison_generate_erroneous_sorting.py` (``..\examples\modules_gallery\comparison\generate_erroneous_sorting.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_comparison_plot_5_comparison_sorter_weaknesses.py` (``..\examples\modules_gallery\comparison\plot_5_comparison_sorter_weaknesses.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_1_recording_extractor.py` (``..\examples\modules_gallery\core\plot_1_recording_extractor.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_2_sorting_extractor.py` (``..\examples\modules_gallery\core\plot_2_sorting_extractor.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_3_handle_probe_info.py` (``..\examples\modules_gallery\core\plot_3_handle_probe_info.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_5_append_concatenate_segments.py` (``..\examples\modules_gallery\core\plot_5_append_concatenate_segments.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_6_handle_times.py` (``..\examples\modules_gallery\core\plot_6_handle_times.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_extractors_plot_2_working_with_unscaled_traces.py` (``..\examples\modules_gallery\extractors\plot_2_working_with_unscaled_traces.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_1_rec_gallery.py` (``..\examples\modules_gallery\widgets\plot_1_rec_gallery.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_2_sort_gallery.py` (``..\examples\modules_gallery\widgets\plot_2_sort_gallery.py``) - - 00:00.000 - - 0.0 From 738dc6a576e7d56fa0413d5df5d225f407ee667f Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Tue, 13 Feb 2024 12:29:37 +0000 Subject: [PATCH 7/8] Integrate the end section of the page into the text. --- doc/how_to/process_by_channel_group.rst | 72 ++++++++----------------- 1 file changed, 22 insertions(+), 50 deletions(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index c5c60ab889..d88f4d3120 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -119,11 +119,29 @@ preprocessed separately per-channel group (then concatenated back together under the hood). It is strongly recommended to use the above structure to preprocess by channel group. -A discussion of the subtleties of the :py:func:`~aggregate_channels` may be found in -the end of this tutorial for the interested reader. -.. warning:: - It is not recommended to apply :py:func:`~aggregate_channels` more than once. +.. note:: + + The splitting and aggregation of channels for preprocessing is flexible. + Under the hood, :py:func:`~aggregate_channels` keeps track of when a recording was split. When + :py:func:`~get_traces` is called, the preprocessing is still performed per-group, + even though the recording is now aggregated. + + To ensure data is preprocessed by channel group, the preprocessing step must be + applied separately to each split channel group recording. + For example, the below example will NOT preprocess by channel group: + + .. code-block:: python + + split_recording = recording.split_by("group") + split_recording_as_list = list(**split_recording.values()) + combined_recording = aggregate_channels(split_recording_as_list) + + # will NOT preprocess by channel group. + filtered_recording = common_reference(combined_recording) + + + In general, it is not recommended to apply :py:func:`~aggregate_channels` more than once. This will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. @@ -168,49 +186,3 @@ Alternatively, SpikeInterface provides a convenience function to sort the record grouping_property='group', working_folder='working_path' ) - - -Further notes on preprocessing by channel group ------------------------------------------------ - -.. note:: - - The splitting and aggregation of channels for preprocessing is flexible. - Under the hood, :py:func:`~aggregate_channels` keeps track of when a recording was split. When - :py:func:`~get_traces` is called, the preprocessing is still performed per-group, - even though the recording is now aggregated. - - To ensure data is preprocessed by channel group, the preprocessing step must be - applied separately to each split channel group recording. - For example, the below example will NOT preprocess by channel group: - - .. code-block:: python - - split_recording = recording.split_by("group") - split_recording_as_list = list(**split_recording.values()) - combined_recording = aggregate_channels(split_recording_as_list) - - # will NOT preprocess by channel group. - filtered_recording = common_reference(combined_recording) - - In the below example the first preprocessing step (bandpass filter) - would be applied by channel group (although, in practice this would have no effect - as filtering is always applied separately to each individual channel). - - However, common referencing (does operate across - separate channels) will not be applied per channel group: - - .. code-block:: python - - split_recording = recording.split_by("group") - - filtered_recording = [] - for recording in split_recording.values() - filtered_recording.append(spre.bandpass_filtered(recording)) - - combined_recording = aggregate_channels(filtered_recording) - - # As the recording has been combined, common referencing - # will NOT be applied per channel group. But, the bandpass - # filter step will be applied per channel group. - referenced_recording = spre.common_reference(combined_recording). From 649b807537c149eae431fb3e5f86ca262b93a305 Mon Sep 17 00:00:00 2001 From: Garcia Samuel Date: Fri, 8 Mar 2024 13:12:29 +0100 Subject: [PATCH 8/8] Update doc/how_to/process_by_channel_group.rst Co-authored-by: Zach McKenzie <92116279+zm711@users.noreply.github.com> --- doc/how_to/process_by_channel_group.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index d88f4d3120..439e757e59 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -114,7 +114,7 @@ to combine the separate channel group recordings back together. combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) Now, when this recording is used in sorting, plotting, or whenever -calling it's :py:func:`~get_traces` method, the data will have been +calling its :py:func:`~get_traces` method, the data will have been preprocessed separately per-channel group (then concatenated back together under the hood).