0.5.1: replace the missing omero metadata text (fix #286)

[joshmoore]

The block of text explaining the requirements of the
"omero" metadata block added in #191 was only applied
to the 0.4/index.bs file and not to latest/index.bs
such that on publication the 0.5/index.bs also did not
have the change.

This restores that text as 0.5.1 with no change to the
on-disk format.

[github-actions]

Automated Review URLs

• Readthedocs
• render latest/index.bs
• diff latest modified

[dstansby]

If a new version is being released, will the version numbers in the ome metadata be updated from 'version': 0.5 to 'version': 0.5.1?

[joshmoore]

If a new version is being released, will the version numbers in the ome metadata be updated from 'version': 0.5 to 'version': 0.5.1?

No.

[dstansby]

So in effect, this means version 0.5 of the specification is being silently changed, without versioning? I appreciate this is fixing a mistake, but doesn't it defeat the point of versioning the spec if specfiic versions aren't frozen after release?

[dstansby]

To give a concrete example, a dataset written with this OME-Zarr metadata is currently valid:

{
  "attributes": {
    "ome": {
      "version": "0.5",
      "omero": {
        "id": 1,
        "name": "example.tif"
      }
    }
  }
}

but if version 0.5 is changed to add

if present it MUST contain the field "channels"

this metadata suddenly becomes invalid because it doesn't contain the "channels" field. If, after this change to the spec is made, a reader sees this metadata, then it's not clear what to do:

1. raise a validation error? That's not possible, because before the OME-Zarr specification was changed someone could have written valid version 0.5 metadata without the "channels" field.
2. not validate that "channels" exists? In that case, the reader is ignoring a (changed) part of the specification, which defeats the point of that part of the specification existing

If instead, the version number is bumped to "0.5.1" in the metadata, this issue is resolved - if a reader sees "0.5", don't validate that "channels" exists. If a reader sees "0.5.1", do validate that "channels" exists.

[joshmoore]

Thanks, David. I appreciate the conscientiousness. Your concerns are certainly valid and in the general case, we want to avoid just that type of breakage, but there's a real-world balance here of managing the complexity for the implementers.

The points which I've taken into account for moving forward with this fix are:

• The fact that the schemas were not also updated, i.e., there is already confusion on which of the two is correct and therefore some correction is needed.
• The release at the Zürich hackathon was "soft", i.e., there has not yet been an announcement.
• Additionally, most of the time during that soft release period was over the holidays and though I know you were working hard both at the hackathon and since on your implementation based on the text of 0.5.0, I don't think that's the general case.
• In fact, for extant implementations, it will be very likely that they will have noticed that the text went missing; it certainly wasn't announced, planned, etc. and therefore they would prefer the 0.5.1 behavior to be restored without the added burden of needed to deal with another version number.

To not allow the incorrect 0.5.0 specification further time to be in the wild, I'm merging this PR. We can begin work on an announcement that explains the situation to the best of our ability.

Thanks again.

[dstansby]

Okay, sounds good, and thanks for actioning quickly!

The release at the Zürich hackathon was "soft", i.e., there has not yet been an announcement.

This wasn't clear to me - what sign can we look for to know if the release is final and not going to change? Just want to make sure we don't waste time implementing an unfinished version over at ome-zarr-models-py, or from a data provider POV I waste time trying to convert data to a non-released version of the spec.

Going forward I think it would be really useful to have the release process written down and codified, so it's clear to folks that a version shown on https://ngff.openmicroscopy.org isn't neccesarily final, and also have some sort of mechanism to mark a release as 'released' (so to speak!)

[joshmoore]

what sign can we look for to know if the release is final and not going to change?

0.5 was the first time we had tried to a "soft" release and I'm not sure I would try it again or even codify it. Though there were pragmatic considerations (hackathon, holidays), to a large extent it was in response to the lack of a reference implementation for Zarr v3. I.e., the community was in a bit of limbo and therefore we were trying to be careful in case some form of emergency change was needed.

Just want to make sure we don't waste time implementing an unfinished version over at ome-zarr-models-py, or from a data provider POV I waste time trying to convert data to a non-released version of the spec.

I would hope we wouldn't even soft release something that would lead to these situations. In this case, the impact I understand it is on the one hand more validation needs adding to o-z-m-p and on the other hand, someone might have started creating non-unvalidatable "omero" blocks but these are the points I covered in #287 (comment)

So, apologies that the text got elided, but I think we're back on track.

Going forward I think it would be really useful to have the release process written down...

Definitely. Let's keep working on that in #282. 👍

[ziw-liu]

Re: soft release: would it be better if there is a formal release candidate phase? Just like most software that have users that the developers don't know would do. Early implementers can then look at the release candidates and report anything that needs to be fixed urgently.

[joshmoore]

@ziw-liu: certainly something that we can also continue on #282 with the caveat that the version number that is embedded in any RC documents will need to also be listed as RC. 👍