Add spot finding and decoding docs #1825

mattcai · 2020-03-12T03:59:51Z

Added tutorials for spot based decoding components and pixel based decoding. Also made minor edits to some of the related classes as well as added ROI to glossary.

…bility.py to accept variable data_set names.

mattcai · 2020-03-12T18:12:24Z

read the docs link for spot finding docs

codecov-io · 2020-03-12T18:16:28Z

Codecov Report

Merging #1825 into master will increase coverage by 0.36%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1825      +/-   ##
==========================================
+ Coverage   87.37%   87.74%   +0.36%     
==========================================
  Files         168      168              
  Lines        5784     5843      +59     
==========================================
+ Hits         5054     5127      +73     
+ Misses        730      716      -14

Impacted Files	Coverage Δ
...sh/core/spots/DecodeSpots/simple_lookup_decoder.py	`100% <ø> (ø)`	⬆️
...e/spots/FindSpots/trackpy_local_max_peak_finder.py	`97.91% <ø> (ø)`	⬆️
...fish/core/spots/FindSpots/local_max_peak_finder.py	`92.61% <100%> (+15.69%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a47383...4b7f077. Read the comment docs.

ttung

can you rebase this on top of the changes you made in #1821?

ttung · 2020-03-12T18:22:23Z

docs/source/creating_an_image_processing_pipeline/index.rst

+
+The spot-based approach finds spots in each image volume based on the brightness of regions
+relative to their surroundings and then builds a :term:`spot trace<Feature (Spot, Pixel) Trace>`
+using the appropriate ``TraceBuildingStrategies``. The spot traces can then be mapped, or


Does this need to link somewhere?

It does now :)

ttung · 2020-03-12T18:23:59Z

docs/source/creating_an_image_processing_pipeline/index.rst

+
+.. _section_which_decoding_approach:
+
+What Decoding Pipeline Should I Use?


so good.

🚀

ttung · 2020-03-12T18:24:35Z

docs/source/creating_an_image_processing_pipeline/index.rst

+If you are unsure which spot finding and decoding methods are compatible with your data here is a
+handy table that summarizes the three major :term:`codebook <Codebook>` designs and what methods
+can be
+used to decode each of them. If your codebook doesn't fall into any of these categories, *make a


link "make a feature request on github" to the issue creation page?

Good idea, done.

ttung · 2020-03-12T18:25:11Z

docs/source/creating_an_image_processing_pipeline/index.rst

 .. _section_segmenting_cells:

 Segmenting Cells
 ----------------

 Unlike single-cell RNA sequencing, image-based transcriptomics methods do not physically separate
-cells before acquiring RNA information. Therefore, in order to characterize cells, the RNA must be
+cells before acquiring RNA information. Therefore in order to characterize cells, the RNA must be


this looks like stuff that changed in your earlier PR.

ttung · 2020-03-12T18:26:26Z

docs/source/help_and_reference/glossary/glossary.rst

@@ -48,6 +48,9 @@ Glossary
        A collection of Image Tiles corresponding to a specific volume or plane of the sample, under which the signal for all channels and all imaging rounds were acquired. All tiles within this FOV are the same size, but the manifest allows for different spatial coordinates for different imaging rounds or channels (to accommodate slight movement between rounds, for example).
        In microscopy, a field of view corresponds to the camera sensor mapped to the sample plane, and many such fields of view are expected to be taken per tissue slice.

+    Region of Interest (ROI)


hm, was this file alphabetically sorted at one point in time?

Maybe? I was debating between ordering it alphabetically vs by complexity. With ctrl-F and links to terms from within the text, nobody needs to search for things alphabetically anymore. And by ordering it from basic to advanced the glossary page is more educational. This is on my to do list before JOSS submission.

ttung · 2020-03-13T05:08:06Z

examples/pixelbased_decoding.py

+important for accurate decoding and it needs vector magnitude and distance thresholds. See
+:ref:`howto_metricdistance` for more details.
+
+After decoding pixels, :py:class:`.PixelSpotDecoder` combines connected  pixels into spots with


Suggested change

After decoding pixels, :py:class:`.PixelSpotDecoder` combines connected pixels into spots with

After decoding pixels, :py:class:`.PixelSpotDecoder` combines connected pixels into spots with

ttung · 2020-03-13T05:08:55Z

examples/how_to/trackpy_local_max_peak_finder.py

+    min_mass=0.02,
+    max_size=2,  # this is max radius
+    separation=7,
+    noise_size=0.65,  # this is not used because preprocess is False


if it's not used, can we not specify this?

I kept it because noise_size is mentioned in the text and is a reminder of what preprocess does. But okay, removed.

ttung · 2020-03-13T05:09:04Z

examples/how_to/trackpy_local_max_peak_finder.py

+    separation=7,
+    noise_size=0.65,  # this is not used because preprocess is False
+    preprocess=False,
+    percentile=10,  # this is irrelevant when min_mass, spot_diameter, and max_size are set properly


This is important to keep imo because it's like a safety measure. I changed "is irrelevant" to "has no effect", do you think that is better?

ttung · 2020-03-13T05:10:43Z

examples/pixelbased_decoding.py

+dpsf.run(imgs, in_place=True)
+glp.run(imgs, in_place=True)
+
+# scale data with user-defined factors to normalize images


Suggested change

# scale data with user-defined factors to normalize images

# scale data with user-defined factors to normalize images. For this data set, the scale factors are stored in experiment.json.

^^ suggested this because it's not required that the scale factors be in experiment.json.

ttung · 2020-03-13T05:12:16Z

examples/pixelbased_decoding.py

+    metric='euclidean', # distance metric to use for computing distance between a pixel vector and a codeword
+    norm_order=2, # the L_n norm is taken of each pixel vector and codeword before computing the distance. this is n
+    distance_threshold=0.5176, # minimum distance between a pixel vector and a codeword for it to be called as a gene
+    magnitude_threshold=1.77e-5, # discard any pixel vectors below this magnitude
+    min_area=2, # do not call a 'spot' if it's area is below this threshold (measured in pixels)
+    max_area=np.inf, # do not call a 'spot' if it's area is above this threshold (measured in pixels)


these are great explanations!

Can we align the comments?

Suggested change

metric='euclidean', # distance metric to use for computing distance between a pixel vector and a codeword

norm_order=2, # the L_n norm is taken of each pixel vector and codeword before computing the distance. this is n

distance_threshold=0.5176, # minimum distance between a pixel vector and a codeword for it to be called as a gene

magnitude_threshold=1.77e-5, # discard any pixel vectors below this magnitude

min_area=2, # do not call a 'spot' if it's area is below this threshold (measured in pixels)

max_area=np.inf, # do not call a 'spot' if it's area is above this threshold (measured in pixels)

metric='euclidean', # distance metric to use for computing distance between a pixel vector and a codeword

norm_order=2, # the L_n norm is taken of each pixel vector and codeword before computing the distance. this is n

distance_threshold=0.5176, # minimum distance between a pixel vector and a codeword for it to be called as a gene

magnitude_threshold=1.77e-5, # discard any pixel vectors below this magnitude

min_area=2, # do not call a 'spot' if it's area is below this threshold (measured in pixels)

max_area=np.inf, # do not call a 'spot' if it's area is above this threshold (measured in pixels)

ttung · 2020-03-13T17:09:17Z

docs/source/creating_an_image_processing_pipeline/index.rst

@@ -235,6 +348,8 @@ segmenting images of stained cells.
 Assigning Spots to Cells
 ------------------------

+* :ref:`Assigning Spots to Cells <tutorial_assigning_spots_to_cells>`


I'm not sure if this belongs or not.

That is just a temporary placeholder.

shanaxel42

This work is really amazing, thank you so much for all the detailed explanations and charts!

shanaxel42 · 2020-03-13T18:22:21Z

examples/how_to/assess_spotfindingresults.py

+about how spots can be found and decoded in starfish see :ref:`section_finding_and_decoding`.
+
+There are two methods for viewing spots. The first is to access the :py:class:`.SpotAttributes`
+of a selected ImageSlice and add it as points to the napari viewer. The second is to use a


It is actually, we call an entry in SpotFiindinigResults PerImageSliceSpotResults

shanaxel42 · 2020-03-13T18:24:19Z

examples/how_to/blob_detector.py

+that can be chosen with the ``detector_method`` argument when instantiating the
+detector. The default LoG approach is best unless computation time becomes a concern.
+
+In order to detects spots of various sizes in the same set of images, :py:class:`.BlobDetector`


Suggested change

In order to detects spots of various sizes in the same set of images, :py:class:`.BlobDetector`

In order to detect spots of various sizes in the same set of images, :py:class:`.BlobDetector`

shanaxel42 · 2020-03-13T18:43:16Z

examples/how_to/tracebuildingstrategies.py

+.. image:: /_static/design/tracebuilder_decisiontree.png
+   :scale: 50 %
+   :alt: Which TraceBuilder To Use
+   :align: center
+


^^^ love a good flowchart

neuromusic

nice job, Matt! a few changes suggested, but looks good to me

neuromusic · 2020-03-13T22:33:12Z

docs/source/creating_an_image_processing_pipeline/index.rst

+using the appropriate ``TraceBuildingStrategies``. The spot traces can then be mapped, or
+*decoded*, to codewords in the codebook using a :py:class:`.DecodeSpotsAlgorithm`.
+
+.. list-table::


will the "width" option prevent the tables from overflowing?

https://docutils.sourceforge.io/docs/ref/rst/directives.html#table

No I tried multiple widths and units, none of them worked. Possibly due to a bug in readthedocs sphinx theme? I know I could fix this by making a grid table. But if we settle on a different sphinx theme that might fix the issue. I've made a note to fix this before JOSS submission.

neuromusic · 2020-03-13T22:35:43Z

examples/how_to/blob_detector.py

+or filters as they are sometimes called, find spots that are the same size as it.
+
+Starfish implements this blob detection technique with :py:class:`.BlobDetector`. It supports three
+`approaches <https://scikit-image.org/docs/dev/auto_examples/features_detection/plot_blob.html>_


not rendering

neuromusic · 2020-03-13T22:58:55Z

examples/spotbased_decoding.py

+"""
+.. _tutorial_spot_based_decoding:
+
+Spot-Based Decoding


since this has no code to render, I think it makes more sense to move this content into the "finding and decoding spots" tutorial, perhaps replacing the "spot finding" section there

I've added code that compares the three spot finding algorithms

…inding how-tos regarding their compatibility with cropped data sets and their support for 2D slices of a volume

neuromusic · 2020-03-20T20:06:48Z

closes #1480

neuromusic · 2020-03-20T20:08:27Z

closes #1749

mattcai added 12 commits February 21, 2020 14:42

added watershed tutorial and fixed typo in MinDistanceLabel docstring

1ec5522

had to remove ':py:param:' because it's not recognized by sphinx

28d5371

fixed broken references

3639d43

fix ':term:<auxiliary images>' not in glossary

132c56f

completed manual segmentation tutorial

19adc96

completed segmentation tutorials and edited ilastik_pre_trained_proba…

b9a326f

…bility.py to accept variable data_set names.

removed 'plot_' prefix and added necessary files

6ec8aae

Merge remote-tracking branch 'origin/master' into mcai-segmentation-docs

97464b9

finished all spot finding and decoding docs

7b6ecd1

undo some changes in notebooks

723ee1e

remove plot_ prefix so it can render in readthedocs

cc27aac

added ROI to glossary and links to it in tutorials

9db96ec

mattcai closed this Mar 12, 2020

Merge branch 'master' into mcai-spots-docs

3f927f8

mattcai reopened this Mar 12, 2020

mattcai requested review from neuromusic, ttung and shanaxel42 and removed request for neuromusic March 12, 2020 18:10

mattcai marked this pull request as ready for review March 12, 2020 18:19

ttung requested changes Mar 13, 2020

View reviewed changes

Revert some unintended changes.

820255f

ttung reviewed Mar 13, 2020

View reviewed changes

shanaxel42 approved these changes Mar 13, 2020

View reviewed changes

neuromusic requested changes Mar 13, 2020

View reviewed changes

addressed starfish team's comments

ecd7cc2

ttung approved these changes Mar 18, 2020

View reviewed changes

added comparison of spot finding methods and added warnings to spot f…

4b7f077

…inding how-tos regarding their compatibility with cropped data sets and their support for 2D slices of a volume

neuromusic linked an issue Mar 20, 2020 that may be closed by this pull request

Finding and Decoding Spots Tutorials #1749

Closed

mattcai requested a review from neuromusic March 20, 2020 20:21

neuromusic approved these changes Mar 23, 2020

View reviewed changes

mattcai merged commit 7cbf539 into master Mar 23, 2020

ttung deleted the mcai-spots-docs branch March 31, 2020 04:08


		.. _section_which_decoding_approach:

		What Decoding Pipeline Should I Use?

	After decoding pixels, :py:class:`.PixelSpotDecoder` combines connected pixels into spots with
	After decoding pixels, :py:class:`.PixelSpotDecoder` combines connected pixels into spots with

	# scale data with user-defined factors to normalize images
	# scale data with user-defined factors to normalize images. For this data set, the scale factors are stored in experiment.json.

	In order to detects spots of various sizes in the same set of images, :py:class:`.BlobDetector`
	In order to detect spots of various sizes in the same set of images, :py:class:`.BlobDetector`

Add spot finding and decoding docs #1825

Add spot finding and decoding docs #1825

Conversation

mattcai commented Mar 12, 2020

mattcai commented Mar 12, 2020

codecov-io commented Mar 12, 2020 • edited Loading

Codecov Report

ttung left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanaxel42 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neuromusic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neuromusic commented Mar 20, 2020

neuromusic commented Mar 20, 2020

codecov-io commented Mar 12, 2020 •

edited

Loading