Support for patched versions - Create "stable" images

[marcbria]

Context

PKP is doing kind of semantic versioning as described by https://semver.org

Syntax is:

major.minor.patch-buildLabel

PKP releases versions with unique git-tags (immutable).
When a security issue or a minor bug is found and fixed, a new tag is generated.
But this is difficult to maintain so, when a new "minor" version is released (ie: 3.3.0-0), last minor is frozen and not updated any more (3.2.1-4).
To keep old versions updated with security patches, PKP is keeping "stable-*" branches.
Those branches ARE updated even for ancient 2.x versions.

Even those changes will be at buildLabel level (for example, stable-3.2.1 would be now tag 3.2.1-5), the problem with docker is those "stable" images would change (two different containers with same name) and this is not a good property in a container (you want to keep the "container immutability" to be sure you are testing/running exactly the same piece of software) so we are talking about different solutions to fix this.

Graphically:

[0]: created because of a security patch.
[1]: updated to include a security patch.
[1]: updated each time a buildLabel release was created
[2]: updated each time a commit occurs to the branch

Where rules are:

• Tags don't change: If a new change is applied, a new tag name is created.
• Branch always matches the last tagged "buildLabel".
• On branches, security patches are applied to all 3.x versions (at x.x.x patch level) and to last 2.x stable.
• On tags, security patches are only applied to the last MAJOR.minor tags.

Solution

We don't have yet a solution for this, but we have some candidates:

a) "Com'on man... is a buildLabel change":
Assume "stable" will be the "latest" version of the branch. Images will mutate, but it will be a "non-braking change" and you will get an easy way to keep your software up to date.

b) "stable-3.2.1-202104101759": Add timestamp to the docker image name.
Versions will be unique, but this naming is confusing for final users. We will need to keep a list of versions to let people find their versions and understand the changes.

c) "stable-3.2.1-5": Follow the syntax with the "buildLabel".
Attach "buildLabel" to each stable release... even when there is no tag with this "buildLabel".
So "3.2.1-4" tag will be the same than "stable-3.2.1-4", and "3.2.1-5" won't never exist but we will get a "stable-3.2.1-5" image.
It approaches assumes PKP will NEVER release a new tag over a frozen branch... other case we will get version-name overlapings.

Additional questions

When a new semantic release is made, does OJS release 3.3.0-0? Do they only release 3.3.0 until there is a security update that warrants a 3.3.0-1?
Yes, they are.

Why not keep the same tag and branch names for the docker tags? (in other words, don’t change the dots to underscores)
Is feasible but PKP asked explicitly to keep our syntax as close as possible to their naming to facilitate the matching.

Is it possible to automatize image names and build?
Daniel suggest to create them automatically at certain time (weekly?) using "postBuild" hook:
https://docs.docker.com/docker-hub/builds/advanced/#custom-build-phase-hooks

[asmecher]

But this is difficult to maintain so, when a new "minor" version is released (ie: 3.3.0-0), last minor is frozen and not updated any more (3.2.1-4).

This is not strictly true -- for example, OJS 3.3.0-3 and 3.2.1-4 were released on the same day. It's just less common for us to release builds from older branches. We want to encourage upgrades and spend our dev efforts on newer work, plus the risk of introducing regressions increases the further we go back, so it takes a pretty high priority problem for us to prioritize work on old branches.

Rather than attaching dates to stable branch names to provide a fixed way to describe the contents of an image when there is not a relevant tag, why not do as git describe? For example, the current stable-3_3_0 HEAD shows:

3_3_0-6

Perfectly clear: there are no commits since that tag. However, stable-3_2_1 shows:

3_2_1-3-13-ga348068d6e

This means that the closest tag is 3_2_1-3, but that there are 13 additional commits, and the latest has tag ga348068d6e. These are unambiguous, human-helpful, and unique.

[marcbria]

Thanks Alec and sorry if I didn't explain the OJS releasing in the right way.
In my defense, say that my intention was not been precise... just showing that, at certain time, old tags are not going to be updated any more.

About docker-image naming... I don't know if I'm following you here Alec.

Are you proposing create tags (aka. names) for docker-images like "3_2_1-3-13-ga348068d6e" (based on git describe that will be unique) instead of "stable-3_2_1" (that change in time)?

If yes, well... I feel like the simpler is the better and this looong name is not easy to remember.

What about trimming it to "3_2_1-3-13" and forget the tag-hash part?

It means we need to adapt our build.sh script to ask for those names, but looks feasible.

But... am I the only one that see the beauty of keeping "stable-3_2_1" as mutable images?
Why I need to edit my docker-composes to add the latest patch name when I can simply stop the container, pull the new image and run the upgrade?

I didn't invent this... projects a serious as nextcloud are offering "latest" images that are ambiguous but facilitate a lot the upgrade work.

What do you think?

[AndrewGearhart]

There's a great story about that describes the troubles of "mutable tags" ... for example, the "latest" tag that is so often included with projects. It seems to make lots of sense until you run into problems when people rely upon it.
TL;DR: Don't use the latest tag when you're doing anything other than "hey, let's throw some spaghetti at the wall to see if it sticks!" Demo? Probably should pin. Instantenous, rebuildable demo? Latest may be okay. Trying something out that nobody will ever look at except you, in this immediate moment? Go ahead... Latest is fine. Production: Oh, god, please, for the children, pin to the correct version.

[asmecher]

I'm thinking we can pretty easily offer whatever variety the user wants...

• If they need a fixed just-as-released-nothing-more, then use the Github tags e.g. 3_3_0-6;
• If they want a mutable image, use the Github stable branches e.g. stable-3_3_0 (caveats above accepted!)
• If they want something in between the two, the output of git describe is a known "standard" (e.g. 3_2_1-3-13-ga348068d6e)

Since all of these can be passed directly to git checkout, can the Docker image not allow the user to choose whichever approach suits them?

@marcbria, for times when we need an unambigous "stamp" but there is no tag available, think 3_2_1-3-13-ga348068d6e is better than 3_2_1-3-13 because the latter probably isn't unambiguous in the git tree, and isn't directly supported by e.g. git checkout.

So for example:

• Production:
  • A production user might start with 3_1_2-0 and work with that for a while. Then they upgrade to 3_1_2-2. So far this is just working with images based on git tags. Very predictable. 👍
  • They get a little behind in upgrades and a security problem is announced. However, because 3.1.x is old, PKP opts not to release a new build.
  • To solve the problem, the user moves from the 3_1_2-0 tag to a later commit on the stable-3_1_2 branch that includes the fix. Per git describe, that would currently be 3_1_2-4-60-g19b453b94b. Still predictable, but doesn't require a github tag indicating a formal release.
• A user assessing OJS for potential use on a test install:
  • A mutable install using the stable-3_3_0 branch is probably what this user would want.

I don't know anything about the Docker release process, so it may be that simply passing through a "commit-ish" to the git commands that set up the code underlying a Docker image isn't possible.

[marcbria]

Still laughing with the spaghetti image. :-D
So, ok Andrew, you won... no "latest" for production any more. ;-)

About Alec's proposal, if nobody is against it... well, I'm not a fan of long hashed names and I suspect Andrew won't be much happy with the "stable-mutants", but we need to move on, so I'm in.

Please, comment or just 👍 the post to be sure we all are ok with this so I can start building the new images.

[AndrewGearhart]

The docker image, like most docker images out there, should offer the ability to override at virtually any point in the construction of the image on it's way to the container. By leveraging the dockerfile (literally just grab the Dockerfile and change what you need to change), a user should be able to use any version of the code. However, supporting this becomes extremely difficult. If we were only talking about the OJS code... it would be pretty simple. How does the collection of plugins get re-versioned to the version that is compatible with the version of OJS that is installed? Maybe I haven't looked at things recently enough... but I can't produce the mental model on how to make the plugins sync up with a by-the-commit version of the software... and would have a hard time imagining that a by-the-commit generation of the images would be permitted by any of the free-as-in-beer image generation locations (github/hub.docker/circleci/etc) ... consider that some of these images are up to 1gb in size... I think we should stick to generating mutable images for branches (ie stable-1_1_1) and particular releases (ie 1_1_1-1). How does that sound?

[marcbria]

You always sound reasonable Andrew. ;-)

But I get lost with your last sentence:

I think we should stick to generating mutable images for branches (ie stable-1_1_1) and particular releases (ie 1_1_1-1). How does that sound?

Alec proposed 3 levels:

• tags (1_1_1-1): non-mutable images (as we are doing right now). Is this what you call "particular releases" isn't it?
• stables (stable-1_1_1): mutable images (probably build weekly?).
• stable-snapshots (3_2_1-3-13-ga348068d6e): non-mutable images (named as each stable is called in git describe).

We arrived to the conclusion that "stable-snapshot" ones are required because, on some regressions, the "tag" level is not generated and only the "stable" is patched.

Are you suggesting ignore the "stable-snapshot" ones?

About the plugins...

I have been also thinking in this problem too.
I had in my toDo adding this issue to gitHub, so if you don't mind, I will do it now to follow the conversation there.

BTW, I discovered this by the hard way... when I accidentally try to ran a ojs2 with my ojs3 plugins in a volume.

But, in short, even I didn't really work over this (only in my brain till now) my first though was also to keep a copy of the plugins with each version (is the "re-version" solution you mention, isn't it?) and I also arrived to the same conclusion than you: To big to host for free, and probably too complex to maintain (what would happen if I reset a container with galley plugins installed or updated after the image building?)...

Anyway, I have a blur sensation that a simpler solution is waiting for us in the corner.

[asmecher]

Just a word about plugin compatibility: The plugin gallery treats compatibility on a per-build level (see http://pkp.sfu.ca/ojs/xml/plugins.xml), i.e. a given plugin needs to have compatibility statements for the specific build in order to be listed in the plugin gallery. In practice, that's more fine-grained than strictly necessary; there are almost no plugins that are compatible with say 3.2.1-1 but not 3.2.1-2. You can always get plugin compatibility information from http://pkp.sfu.ca/ojs/xml/plugins.xml, and if you think there might be something helpful we can do to inform the Docker image of compatibilities, that might be a discussion worth having.