Add test OLM upgrade path

[eguzki]

What

Implements #1117 and is part of the work being done for #1112

It's a Github Action Workflow to test the Kuadrant upgrade process using OLM APIs.

It is being triggered manually (on.workflow_dispatch) and accepts as input:

• kuadrantStartVersion: Kuadrant start version (required). For example: 1.0.0 (without v prefix)
• kuadrantNamespace: Namespace where Kuadrant is installed (optional defaults to kuadrant-system).
• clusterServer: Cluster server URL (optional defaults to local kind cluster). For example: https://example.com:443
• clusterToken: Cluster Server Bearer Token (optional defaults to local kind cluster token). For example: sha256~abcdefghijk12345432432

The workflow goes as follows:

• It Installs Kuadrant release with the version specified by kuadrantStartVersion. It's being installed traditionally using OLM: OperatorGroup + CatalogSource (with the catalog build for the kuadrantStartVersion) + Subscription.
• Waits for subscription, CSV and deployments to be ready
• Deploys kuadrant CR and checks status is Ready. The kuadrant operator, runs a set of tests to mark the resource status as ready, so effectively, the workflow is relying on those tests to verify installation.
• When the starting point installation is ready, it proceeds with the upgrade process. The upgrade is being carried out running operator-sdk run bundle-upgrade command. The command, basically, builds an ephemeral index adding an upgrade path from the existing installed bundle to the given bundle as input param. Then, waits until all deployments are ready (with a timeout set to 3 mins) and fails if some deployment is not ready after the timeout expires. Upgrade may fail for instance if some referenced docker image is not valid or does not exist. The upgrade verification steps by the workflow relies, once again, on 1) upgraded CSV readiness, 2) upgraded deployment image and 3) the kuadrant CR status. For example, if for whatever reason, let's say the limitador operator, or limitador instance, is not up and running, kuadrant CR would report on the status and the upgrade test would fail. The upgrade command may fail if kuadrant dependencies cannot be upgraded as well. This issue is not obvious from the upgrade command output, so digging is required.

Simple commands are being coded as part of the Github workflow. For high level commands, those are implemented using makefile targets or third party tools so they can be easily run locally to dev/test.

Verification Steps

Prepare a new release, let's say v1.0.2

git checkout -b testing-upgrade
make prepare-release VERSION=1.0.2 AUTHORINO_OPERATOR_VERSION=0.16.0 LIMITADOR_OPERATOR_VERSION=0.12.1 DNS_OPERATOR_VERSION=0.12.0 WASM_SHIM_VERSION=0.8.1 RELATED_IMAGE_CONSOLEPLUGIN=quay.io/kuadrant/console-plugin:v0.0.18

Commit and push the branch (testing-upgrade in the example).

Then, run workflow manually (cannot be done from github UI as it has not been merged to main branch). This run would test upgrade from v1.0.1 to v1.0.2:

gh workflow run test-upgrade.yaml --ref testing-upgrade -f kuadrantStartVersion=1.0.1

Check the upgrade test workflow run to complete successfully.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.42%. Comparing base (318fd81) to head (b06d471).
Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1165      +/-   ##
==========================================
+ Coverage   83.30%   83.42%   +0.11%     
==========================================
  Files          81       81              
  Lines        6943     6943              
==========================================
+ Hits         5784     5792       +8     
+ Misses        933      925       -8     
  Partials      226      226              

Flag	Coverage Δ
bare-k8s-integration	22.83% <ø> (+0.11%)	⬆️
controllers-integration	74.91% <ø> (+0.23%)	⬆️
envoygateway-integration	40.87% <ø> (+0.42%)	⬆️
gatewayapi-integration	19.63% <ø> (+0.04%)	⬆️
istio-integration	43.45% <ø> (+0.14%)	⬆️
unit	19.57% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	90.00% <ø> (ø)
api/v1beta2 (u)	∅ <ø> (∅)
pkg/common (u)	∅ <ø> (∅)
pkg/istio (u)	62.06% <ø> (ø)
pkg/log (u)	93.18% <ø> (ø)
pkg/reconcilers (u)	24.67% <ø> (ø)
pkg/rlptools (u)	∅ <ø> (∅)
controllers (i)	86.76% <ø> (+0.19%)	⬆️

see 8 files with indirect coverage changes

[Boomatang]

I tried running the verification steps and it failed.
https://github.com/Kuadrant/kuadrant-operator/actions/runs/13286208550/job/37095372107

[eguzki]

I tried running the verification steps and it failed. https://github.com/Kuadrant/kuadrant-operator/actions/runs/13286208550/job/37095372107

You tried to upgrade kuadrant from 1.0.1 to 0.0.0. Not surprised it failed. Did you prepare a new release?

[Boomatang]

I copied and pasted the instructions in the verification steps.

[eguzki]

I copied and pasted the instructions in the verification steps.

Your workspace in the pushed commit does not have the make/release.mk that is generated with make prepare-release target.

❯ git checkout c0a85bbed5f05268ff474a273a062a0dfb06a349
HEAD is now at c0a85bbe Workflow testing
❯ cat make/release.mk
cat: make/release.mk: No such file or directory

Might be you did not add all the changes to the commit. Could it be?

[maleck13]

I copied and pasted the instructions in the verification steps.

Your workspace in the pushed commit does not have the make/release.mk that is generated with make prepare-release target.

❯ git checkout c0a85bbed5f05268ff474a273a062a0dfb06a349
HEAD is now at c0a85bbe Workflow testing
❯ cat make/release.mk
cat: make/release.mk: No such file or directory

Might be you did not add all the changes to the commit. Could it be?

I This worked great for me using the verification steps. I did have to add the make/release.mk file manually as it wasn't added as part of the prepare release command.
https://github.com/Kuadrant/kuadrant-operator/actions/runs/13365901249/job/37323516174

[maleck13]

LGTM