Releaseworkflow

[markusweigelt]

Unfortunately, I still have some ambiguities regarding the release workflow and for this reason I would like to discuss them to avoid misunderstandings.

Here briefly description as I understood you:

The CI workflow makefile-ci.yml in the main branch of our ocrd_kitodo repository has run successfully after the last push. Now we decide to make a new release. For each repository OCR-D Controller, OCR-D Manager and OCR-D Monitor an image with the tag stable should be built now. This is done according to your statement by a manual triggering e.g. by a stable branch which receives the new version of the master or main with a push and builds the image via a corresponding workflow.

After that, the stable branch of ocrd_kitodo is upgraded by merging the current state of main branch into it. This triggers a workflow that replaces all necessary .env image tag names with stable, prepares and runs make start to pull the stable images from the repositories. After the successfully test run with make test the release is created manually.

Is it as you have described this process or have I misunderstood something here?

In my opinion, this release workflow is inconvenient and also misleading. Inconvenient because here are lot of manual tasks have to be executed on different repositories. Misleading cause if I clone the stable branch of the base repository and build with make build I maybe get a different version of the respective image than if I pull it directly.

My goal is to have as little manual effort as possible to do the release and the last stable state should be independent of using make build or pull the stable tag of image directly.

The way I would like the workflow to be

The CI workflow makefile-ci.yml in the main branch of our ocrd_kitodo repository has run successfully after the last push. Now we decide to make a new release. We merging the state of main branch into the stable branch of our ocrd_kitodo repository. After the makefile-ci.yml workflow passed successfully we ...

Variant 1
... deploy the builded and tested images to the respective repository with the stable tag

Variant 2
... start external workflow e.g. update-stable.yml of submodule repository using workflow dispatch with corresponding submodule "commit as ref" as param (https://github.com/orgs/community/discussions/26323#discussioncomment-3251454).

Variant 2.1
The update-stable.yml workflow create a runner checkout with "commit as ref" and build and deploy the new stable image version to the contrainer registry.

Variant 2.2
The update-stable.yml workflow update the stable branch and the new stable image version is created by a separate workflow when stable branch changed.

After that we create the new release manually using the stable branch of ocrd_kitodo.

fyi @bertsky @SvenMarcus

[bertsky]

Here briefly description as I understood you:

The CI workflow makefile-ci.yml in the main branch of our ocrd_kitodo repository has run successfully after the last push. Now we decide to make a new release. For each repository OCR-D Controller, OCR-D Manager and OCR-D Monitor an image with the tag stable should be built now.

Yes, but only for those submodules which need to be updated (from their last stable) do we need a rebuild.

This is done according to your statement by a manual triggering e.g. by a stable branch which receives the new version of the master or main with a push and builds the image via a corresponding workflow.

Yes, so basically, for any submodule that you want to give a new stable, you do a git -C _modules/MODULE push origin HEAD:stable locally. Or on Github by clicking on that branch, comparing it to master, opening a pull request and merging master into stable. That's all you need to do manually.

After that, the stable branch of ocrd_kitodo is upgraded by merging the current state of main branch into it.

Yes. Again, this can be done locally (via git checkout stable; git merge main; git push origin stable) or on Github (via PR).

This triggers a workflow that replaces all necessary .env image tag names with stable,

No. The :stable image tags are already there in .env – they never go away on the stable branch.

Every merge should usually keep this (because it's unlikely there are conflicts here – main will stay at :latest, stable will stay at :stable). Only the submodule revisions (and compose files, Makefile or whatever) is going to be updated during merges.

The CI workflow itself, which gets triggered by the push, does not change any files.

prepares and runs make start to pull the stable images from the repositories.

Yes, and because these images get pulled from GHCR on the submodules, we must wait for all submodules to be rebuilt before pushing/merging into ocrd_kitodo:stable.

After the successfully test run with make test, ...

Yes, the CI workflow then picks it up and runs make test.

... the release is created manually.

We could now either create a new release on the Github UI, or have the (successful) CI do that automatically. (For example, in ocrd_all's release.sh, we aggregate the changelogs/commit messages from all submodules since the last release.)

Is it as you have described this process or have I misunderstood something here?

IIUC you did misinterpret some details, from which the overall picture might have looked too bleak. (But perhaps I just misread your formulations.)

In my opinion, this release workflow is inconvenient and also misleading.

Inconvenient because here are lot of manual tasks have to be executed on different repositories.

The only manual steps I can see are those which we would never want to be automatic: triggering new stable versions in submodules (and only those you want to update), and triggering a new stable version at the top. That triggering can happens on a local git clone or in Github itself.

Misleading[, be]cause if I clone the stable branch of the base repository and build with make build I maybe get a different version of the respective image than if I pull it directly.

No. If you clone and checkout the stable branch, then each module will be at the same commit which was used to build its most recent :stable image on Github, and the top-level .env will point to these images. So both make build and make pull should yield the same result.

(But of course the whole point of publishing stable images is so you don't need to rebuild locally, so I would not even call it misleading if there actually was a difference. A checkout is only necessary because the images themselves can only be run along with their docker-compose.yml and the .env file. Rebuilding Docker images is a developer task, not a user action.)

My goal is to have as little manual effort as possible to do the release and the last stable state should be independent of using make build or pull the stable tag of image directly.

Both is the case here IMO.

The way I would like the workflow to be

The CI workflow makefile-ci.yml in the main branch of our ocrd_kitodo repository has run successfully after the last push. Now we decide to make a new release. We merge the state of main branch into the stable branch of our ocrd_kitodo repository [and push it – unless already on Github].

Ok, so at this point we have a new commit on ocrd_kitodo:stable, but nothing new on the submodules yet.

[After the makefile-ci.yml workflow passed successfully] we ...

Variant 1 ... deploy the built and tested images to the respective repository with the stable tag

… which means giving up the modules' responsibility for their own packages – everything gets deployed from ocrd_kitodo (with some key sharing magic involved).

@SvenMarcus you had some reservations against managing packages from outside their respective repos IIRC?

So IIUC you would rewrite makefile-ci.yml such that it applies to both main an stable, is responsible both for testing and eventually for deployment (of all submodule images) in both cases, just uses :latest in one and :stable in the other?

I actually do like the simplicity here. (Once we have set everything up of course.)

Variant 2 ... start [an] external workflow e.g. update-stable.yml of [each] submodule repository using workflow dispatch with [the] corresponding submodule commit as ref as param (https://github.com/orgs/community/discussions/26323#discussioncomment-3251454).

Yes, that would be even better. We still need to share the GITHUB_TOKEN across repos, but at least the build itself would be local to each repo.

And again, this could apply to both :latest and :stable (depending on ocrd_kitodo and passed as another param).

Variant 2.1 The update-stable.yml workflow[s] create a runner checkout with "commit as ref", and build and deploy the new stable image version to the contrainer registry.

Variant 2.2 The update-stable.yml workflow[s] update the stable branch [via "commit as ref"] and the new stable image version is created by a separate workflow when [the] stable branch changed.

I do like this better – the image states get reflected by the repo branches.

Also, I believe this would be fully compatible with the manual triggers described by the method proposed originally.

After that we create the new release manually using the stable branch of ocrd_kitodo.

Again, we could write a release script that aggregates all changes textually here.

[SvenMarcus]

I agree with that and I also like variant 2.2 for handling stable updates the best.

[markusweigelt]

Every merge should usually keep this (because it's unlikely there are conflicts here – main will stay at :latest, stable will stay at :stable). Only the submodule revisions (and compose files, Makefile or whatever) is going to be updated during merges.

Ok then let's see how it works. The process does not happen so often and then we will just worry if it leads to problems.

No. If you clone and checkout the stable branch, then each module will be at the same commit which was used to build its most recent :stable image on Github, and the top-level .env will point to these images. So both make build and make pull should yield the same result.

Exactly I did not assume that we put the submodules on the stable branches as well. That was my thinking error.