Image segmentation using custom neural network model with Tensorflow

[gusqgm]

Currently two different network architectures have been used for object segmentation: Cellpose and RDCNet. As a minimal working example, we should be able to use a pre-trained model from each of these architectures and use it for prediction of label maps. Important to note that Cellpose is using pytorch, whereas RDCNet utilizes Tensorflow.

Cellpose should work by installing it from the repository above. For RDCNet, we have usually used it via ISIT.

The input is the reading of the zarr file in such a way that the images match the images used for training and applying the prediction model on them. The output are object labels which are defined as regions which contain information of where they should be located within the field grid. Each region comprises of a segmented object plus a certain bounding box surrounding it.

Currently the models mostly used with RDCNet rely on object segmentation of MIP projections of each well/overview.

[gusqgm]

In a more organized fashion:

1. input zarr files manipulate the images so they fit the trained images for the segmentation model. For example, making maximum intensity projections of each well overview
   1.2 Objects are predicted using either one of the networks
2. outputs saved as regions within the overview grid: an object is a bounding box region with a mask
   2.1 segmented object with certain boundary surrounding it (2D or 3D)
   2.2 each object gets assigned the grid numbers where its bounding region overlaps
   2.3 list of all objects and their grids saved in table format
   2.4 if segmentation occurs at multiple levels (e.g. tisse, cells, subcellular structures), smallest object(s) is(are) linked to its(their) parent via particular label mapping

[gusqgm]

Here an interesting challenge to check is whether the construct of bounding box + mask can be used as a generic unit that e.g. Napari could use for dynamically reading only the pertaining chuncks of the larger .zarr files for display. this could be particular interesting in situations where the user can also filter out particular objects based on certain values, so that only the valid objects are loaded into the canvas. Something going in the direction of the Napari Clusters Plotter

[tcompa]

Can we move this discussion to fractal-analytics-platform/fractal-client#64 and close this one, or are there important differences?

Also: I don't have access to https://github.com/fmi-basel/isit (in case it's useful).

[jluethi]

@tcompa There is quite some overlap. I now renamed it to reflect its difference:
fractal-analytics-platform/fractal-client#66 is about the generic implementation of an existing neural network in Pytorch with a clear python API and no known dependency issues (=> cellpose).

Let's use this issue to track progress on the more complex case of some custom models at FMI. This is not a July goal. I think the first step will be some cleanup on those models and the isit library that runs them. Once this is achieved, we can integrate the more complex network architectures into Fractal
(=> this is a backlog issue for the moment, @gusqgm will update us here once we have progress on the FMI side about the networks)

[jluethi]

This has been covered in the (private) RDCNet task at FMI now