Improve interactive OME-Zarr viewing performance in napari

[jluethi]

Loading large, multi-well OME-Zarr projects into napari and browsing them interactively is still on the slower side. Here is an overview of what we're doing to improve that and what we can still try.

Things that we have already found to improve this:

• Setting environmental variable NAPARI_ASYNC=1 before launching napari: Loading still a bit slow, but interface doesn't freeze while loading
• Having the zarr files use compression (was already on by default), see here: Enable compression of Zarr files #28

Things we are working on:

• Errors thrown about non-existing wells, see issue here: Large-scale ome-zarr errors when dragged into napari #35
• Having more pyramid levels, see here: Generalize pyramid creation #32. Currently, our lowest-resolution pyramid-level is still 1215 x 1280 per well. Looking at some example datasets, e.g. this one from the ome developers, they have pyramid levels as small as 67x67 per site (also sites saved individually, see below in Zarr organisation) => This currently doesn't help at all, investigating

Remaining issues:

• What pyramids are loaded when. During initial loading, the full resolution tile seems to be loaded (to calculate rescaling parameters maybe?). See here for the logs: Large-scale ome-zarr errors when dragged into napari #35. Also, while browsing, napari sometimes quickly switches between two pyramid levels (e.g. between level 3 & 4).
• The "aggressiveness" of napari loading new pyramid levels is also mentioned by the napari-ome-zarr plugin devs here: Plate pyramids and version ome/ome-zarr-py#195

Potentially contributing:

• The Zarr organisation: Do we split multiple field of views or save all fields of view of a well as a single field of view, see here: Is the OME hierarchy correct? Field of view placement in hierarchy #36 => Unlikely to be an issue. According to OME devs, our implementation may actually a bit faster, even if it's not fully as intended

[jluethi]

I compared our performance with that for a test dataset that the ome-zarr devs use here:
ome/ome-zarr-py#195
napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/422.zarr -vvv
(does not work with our version of the plugin, but if one installs the branch of that pull request, it works)

It opens faster (lower-res pyramids available), but browsing is fairly similar to our performance, so I would say that our OME hierarchy probably isn't the major factor slowing us down, it may come down mostly to the speed of the connection and the aggressiveness with which napari loads pyramid levels (and the apparently very limited caching of such levels it seems to do, e.g. zooming out typically needs to load the layers freshly).
They also don't have missing wells, so unclear whether that is an important influence

[jluethi]

Having more pyramids should make initial loading faster, right? No!
I went from 5 levels of pyramids (initial loading of plate ~ 4 min via VPN) to 10 levels of pyramids (initial loading of plate ~25 min via the same VPN connection)
=> looks like there are issues with too many pyramid levels or too many directories or such when loading ome zarr files into napari.
I've installed a dev version of the ome-zarr plugin (with their ome/ome-zarr-py#195 pull request for pyramid loading) and will start profiling this to get an idea where the issue could be (/understand what's going on).

[jluethi]

Improved chunking (see here: #32) massively improves interactive performance. We went from a chunking scheme where every original field of view had its own pyramid, thus there were the same number of files on every pyramid level. This took some 3+ minutes to load the plate and zooming was quite slow.

Now we have rechunked pyramids, such that the chunks stay approximately constant in size and thus there are fewer and fewer files, the lower the resolution of the pyramid becomes. This loads the 23 well plate in ~15s!
Also, interactive zooming seems somewhat improved. It's still quite slow to load full-res images, but faster than before.

Learnings:
Having tiny chunks is a performance hit in interactive browsing. Also, wonder how well the interactive browsing in general will scale with the number of files in the zarr container.

[jluethi]

We just talked with Kevin Yamauchi today (napari core dev). There is actually a large restructuring of the whole lazy-loading process currently in the works in the napari team, with Andy Sweet in the lead apparently. We'll try to attend some relevant napari community meetings to show our use case and get some ideas on whether they already work on things that will make our life easier :)

[tcompa]

Useful refs:

• https://github.com/andy-sweet/napari/blob/55b025a6013f90914f15bca7030aca98ed542b40/docs/naps/4-async-slicing.md
• [WIP]: NAP-4 async slicing prototype  napari/napari#4905

[jluethi]

Yes, the PR implementing these changes improves visualization performance quite a bit. It doesn't change the actual loading speeds that much, but makes the interface fully responsive during all movement and thus improves the experience of browsing the dataset. Had a few rounds of tests & improvements with Andy Sweets to get rid of some bugs, very optimistic about this implementation now.
See here: napari/napari#4905

I think it's worth using that PR when we want to test performance and it may make for better demos :)

I'd say we can close this for the time being. Let's wait for the PR to make its way into napari and for sharding to come to Zarr to improve the performance further :)
https://forum.image.sc/t/storing-large-ome-zarr-files-file-numbers-sharding-best-practices/70038