Working with multi channel images

[tischi]

Sorry, for putting so many issues!
It's a lot of fun to use the assistant! Great concept!

I am confused by working on a two channel image. I was using Extract channel two times within the workflow, but then I cannot see those steps in the recorded workflow (see below). The funny thing though is that the workflow seems to run correctly, but I don't understand how from reading the code. I assume the channel splitting happens during the // Copy steps?

// To make this script run in Fiji, please activate 
// the clij and clij2 update sites in your Fiji 
// installation. Read more: https://clij.github.io

// Generator version: 0.4.2.24

// Init GPU
run("CLIJ2 Macro Extensions", "cl_device=");

// Load image from disc 
open("/Users/tischer/Downloads/Tischi-RPbodies/Exp9-01(4).czi");
image1 = getTitle();
Ext.CLIJ2_pushCurrentZStack(image1);
// The following auto-generated workflow is made for processing a 2D or 3D dataset.
// For processing multiple channels or time points, you need to program a for-loop.
// You can learn how to do this online: https://www.youtube.com/watch?v=ulSq-x5_in4

// Copy
Ext.CLIJ2_copy(image1, image2);
Ext.CLIJ2_release(image1);

Ext.CLIJ2_pull(image2);

// Sum Z Projection
Ext.CLIJ2_sumZProjection(image2, image3);
Ext.CLIJ2_release(image2);

Ext.CLIJ2_pull(image3);

// Copy
Ext.CLIJ2_copy(image3, image4);

Ext.CLIJ2_pull(image4);

// Threshold Otsu
Ext.CLIJ2_thresholdOtsu(image4, image5);

Ext.CLIJ2_pull(image5);

// Binary Fill Holes
Ext.CLIJ2_binaryFillHoles(image5, image6);
Ext.CLIJ2_release(image5);

Ext.CLIJ2_pull(image6);

// Connected Components Labeling Box
Ext.CLIJ2_connectedComponentsLabelingBox(image6, image7);
Ext.CLIJ2_release(image6);

Ext.CLIJ2_pull(image7);
run("glasbey_on_dark");

// Extend Labels With Maximum Radius
radius = 20.0;
Ext.CLIJx_extendLabelsWithMaximumRadius(image7, image8, radius);
Ext.CLIJ2_release(image7);

Ext.CLIJ2_pull(image8);
run("glasbey_on_dark");

// Copy
Ext.CLIJ2_copy(image3, image9);
Ext.CLIJ2_release(image3);

Ext.CLIJ2_pull(image9);
Ext.CLIJ2_release(image9);

// Label Mean Intensity Map
Ext.CLIJx_labelMeanIntensityMap(image4, image8, image10);
Ext.CLIJ2_release(image4);
Ext.CLIJ2_release(image8);

Ext.CLIJ2_pull(image10);
run("glasbey_on_dark");
Ext.CLIJ2_release(image10);

[haesleinhuepf]

Sorry, for putting so many issues!

Feedback is ❤️ welcome! Thanks for guiding clij development!

I am confused by working on a two channel image. I was using Extract channel two times within the workflow, but then I cannot see those steps in the recorded workflow (see below). The funny thing though is that the workflow seems to run correctly, but I don't understand how from reading the code. I assume the channel splitting happens during the // Copy steps?

Correct, as clij has no concept of channels, it cannot have a command such as "extract channel". The comment above is hinting you towards using a for-loop if you want to process all channels. I'm still looking for the right way of dealing with this because the "extract" channel step must in macro happen before the push and thus, at a different place in the code actually. In practice, you should do something like the following workaround at the very beginning before processing any image:

Stack.setChannel(1);
rename("channel1");
Ext.CLIJ2_pushCurrentZStack("channel1");

Stack.setChannel(2);
rename("channel2");
Ext.CLIJ2_pushCurrentZStack("channel2");

If you have any ideas how to make this easier, my rabbit ears are wide open.

Have a great weekend and thanks again for the feedback!

Cheers,
Robert

[tischi]

What about just not supporting multi-channel images at all? This would be quite radical, but it would also simplify many things. For example in BDV, the image data model is a Source which is only 3D+t. If you want multiple channels you need multiple sources.

So, in CLIJ when people do their first operation on an 5D stack, there could be something happening that forces the user to split the image into multiple 4D stacks. Another example is MorpholibJ, which, afiak, also does not work on multi-channel images (ping @dlegland).

[dlegland]

Hi!
I confirm, in MorphoLibJ we do not work with multi-channel images.

I agree with @tischi, most of the time it is better to work on channels individuallay. When the information is different, then the processing often is.

[haesleinhuepf]

What about just not supporting multi-channel images at all?

Hehe, that was my first approach. End-users kept asking for "How about multi-channel data and time-lapses?"

So, in CLIJ when people do their first operation on an 5D stack, there could be something happening that forces the user to split the image into multiple 4D stacks.

Before implementing this, we need to classify operations. For example, cylinder and maximum projections, affine transforms, intensity corrections, drift corrections do make a lot of sense on multi-channel data. Furthermore, I haven't implemented anything yet that makes use of multi-channel information such a pixel classifiers...

I agree with @tischi, most of the time it is better to work on channels individuallay. When the information is different, then the processing often is.

I also agree. And most users use Extract channel at the correct place in their workflows. The question might rather be how to implement the correct handling of multi-channel data in the code-generators :-)

[haesleinhuepf]

Just an example, assume that operations A, B and C are applied to multi-channel data of channels 1, 2 and 3. This order of execution

A1-A2-A3-B1-B2-B3-C1-C2-C3

Is potentially a tiny bit faster and but needs three times more memory in the GPU than this order:

A1-B1-C1-A2-B2-C2-A3-B3-C3

Would be cool if the code-generator could generate code accordingly, maybe depending on users preferred speed/memory resource handling.

[tischi]

This is too deep for me to comment without a little hackathon :-)

What would already help a lot is to improve this also for the multiple channels:
If the names in the recorded scripts would give the reader a chance to see to which channel the images belong to it would be already much better.

[tischi]

I tried again with the new version and the channel extraction is still elusive to me (all those "copy" commands"). And, in fact, I think it does not work.

When I recorded the script, it extracted the third channel (nuclei), but when I ran the script if extracted the first channel.

// To make this script run in Fiji, please activate 
// the clij and clij2 update sites in your Fiji 
// installation. Read more: https://clij.github.io

// Generator version: 0.4.2.25

// Init GPU
run("CLIJ2 Macro Extensions", "cl_device=");

// Load image from disc 

// TISCHI: Manually commented out (but does not work either)
//open("/var/folders/91/6n93mtg108g6161n3sx_x33m0000gn/T/temp1617980102305.tif");

run("Fluorescent Cells"); // TISCHI: manually added

result_image_1 = getTitle();
Ext.CLIJ2_pushCurrentZStack(result_image_1);
// The following auto-generated workflow is made for processing a 2D or 3D dataset.
// For processing multiple channels or time points, you need to program a for-loop.
// You can learn how to do this online: https://www.youtube.com/watch?v=ulSq-x5_in4

// Copy
Ext.CLIJ2_copy(result_image_1, result_copy_2);
Ext.CLIJ2_release(result_image_1);

Ext.CLIJ2_pull(result_copy_2);

// Copy
Ext.CLIJ2_copy(result_copy_2, result_copy_3);
Ext.CLIJ2_release(result_copy_2);

Ext.CLIJ2_pull(result_copy_3);

// Threshold Otsu
Ext.CLIJ2_thresholdOtsu(result_copy_3, result_threshold_otsu_4);
Ext.CLIJ2_release(result_copy_3);

Ext.CLIJ2_pull(result_threshold_otsu_4);

// Connected Components Labeling Box
Ext.CLIJ2_connectedComponentsLabelingBox(result_threshold_otsu_4, result_connected_components_labeling_box_5);
Ext.CLIJ2_release(result_threshold_otsu_4);

Ext.CLIJ2_pull(result_connected_components_labeling_box_5);
run("glasbey_on_dark");
Ext.CLIJ2_release(result_connected_components_labeling_box_5);

[haesleinhuepf]

I think it does not work.

I know. See my comment above:

Correct, as clij has no concept of channels, it cannot have a command such as "extract channel". The comment above is hinting you towards using a for-loop if you want to process all channels. I'm still looking for the right way of dealing with this