Update standard K8s version policy

[cah-hbaum]

The standard for the K8s version policy is updated in order to add the support period for Kubernetes versions into it. Since this is a bit contradictive with the K8s version recency, the whole standard needed to be updated.

[garloff]

Two points on language:

• Standards should not make extensive use of "probably" or "we believe". We need to clarify these things until we are convinced. And then just make statements what is expected.
• relative clause "The reason that XYZ can be done is ..." do not have commas in English

Two on the content:

• This has been toned down into mere recommendations while the original intention was to have stronger language MUST. Did we agree as a team to really make it so soft? How much value remains then if customers can not rely on anything even if the container SCS-compatible cert has been granted?
• There seems to be confusion about support period for minor versions that puzzles me. Let's clarify this!

[cah-hbaum]

Answering to #334 (review) and #334 (comment) since they ask the same things:

So I had a misunderstanding on how this K8s providing should work, since I thought that would be the provided version for things like managed K8s clusters.
If the K8s version means the version made available to the users of a cloud by the provider, this would imply two problems to me:

1. The previously made test for k8s version recency is wrong, since we just test a version of a cluster. This wouldn't matter in the case described above, since a cluster wouldn't need to be updated right away (only the version the provider needs to offer) and the failure on the test if a version is delayed (according to the standard) would be false in this case as well.
2. If a new version is published by Kubernetes, I see no reason to not offer it right away. I think the versions are sufficiently tested and having a delay between the official release date and a "provided" release date just feels not right.

I think what I would want to know here to clear that up:
What is the use-case?
How are the Kubernetes versions provided?
And can this even be tested (in the next step), since we can't check if a version is "provided", only the version of a cluster (?)

I would continue here if these things are cleared up, since I think they're essential to continuing.

[garloff]

So I had a misunderstanding on how this K8s providing should work, since I thought that would be the provided version for things like managed K8s clusters.

I assume that even in managed k8s offerings, the k8s version updates are normally done only with the user's consent.
At least for minor updates (like going from v1.24.xx to v1.25.xx), the upgrade tends to require the user (who manages the workloads) to review the changelogs with the deprecations ... to ensure the workload is still working after such an upgrade.

So, looking at a cluster and determine its level of outdatedness may be useful, but does not provide important insight into standards compliance. (Well, if it is recent, we may be able to conclude that the operator is SCS-compatible, but the outdatedness does not mean that he's not.)

Instead, we need to ensure that the latest patchlevel (with a delay of less than a week) is available.
Same for latest minor versions (with a max delay of a generous 4 months).

I agree that testing this is difficult.

• In our reference implementation, users can select v1.27.x and get v1.27. in return.
  For this, we need to adjust https://github.com/SovereignCloudStack/k8s-cluster-api-provider/blob/65af7217a4c2d849d1a408d36a3c9c6ea6c554aa/terraform/files/bin/openstack-kube-versions.inc#L6
  and the node image must be available. This one comes from OSISMs minio:
  https://github.com/SovereignCloudStack/k8s-cluster-api-provider/blob/65af7217a4c2d849d1a408d36a3c9c6ea6c554aa/terraform/files/bin/upload_capi_image.sh#L30
• If we fail to touch the openstack-kube-versions.inc file, the user could still successfully force a version. As long as the node image is available, this will work.
• Some providers might serve their customers well by registering those node images centrally as opposed to having each cluster user download them as needed. (The scripts will use those pre-registered images then.)

Now, this is all specific to our reference implementation, unfortunately.
Tests can be developed to check openstack-kube-versions.inc and the image availability.
In a generic approach, we have not defined a way to discover available k8s versions, so we can not yet cover other providers. Maybe we'll have a mechanism for all clusterstack users when we move to next-gen Ref.Impl.

[cah-hbaum]

Now, this is all specific to our reference implementation, unfortunately. Tests can be developed to check openstack-kube-versions.inc and the image availability. In a generic approach, we have not defined a way to discover available k8s versions, so we can not yet cover other providers. Maybe we'll have a mechanism for all clusterstack users when we move to next-gen Ref.Impl.

Ok I think I understand the intention behind all of this, but doesn't this still make the first test for k8s cluster recency incorrect, since the test is only applied to a cluster and we shouldn't look for a version recency in a cluster, since this is seemingly up to the user?

[cah-hbaum]

Made an update to the text.
I still think it will be very hard to test this (if its even possible), but we will see. This is only the standard for now.

[anjastrunk]

Suggested change

	**3 minor** releases per year [Kubernetes Release Cycle].
	**3 minor** releases per year [Kubernetes Release Cycle](https://kubernetes.io/releases/release/).

[anjastrunk]

I just recognized, that links are listed at the end of the document. Is this a common practice in SCS standards?

[garloff]

No. Linking docs that explain a statement or term is best be done in place. A link I would click if I need an explanation or definition or want to see the source of truth for a statement.

A references section at the end is more for further reading or background reading (in case I'm new to the topic).
Also link to related content, e.g. related standards.

[cah-hbaum]

Should we really put links in different locations depending on their relation to the standard? I would rather try to put everything in the final section and check if the link nevertheless works in the document itself.

[anjastrunk]

In favor of readability, I would add links in place. And put, as @garloff mentioned, additional documents, not related to a special term, at the bottom of the document.

[cah-hbaum]

Put the links in the text.

[garloff]

If the K8s version means the version made available to the users of a cloud by the provider, this would imply two problems to me:

1. The previously made test for `k8s version recency` is wrong, since we just test a version of a cluster. This wouldn't matter in the case described above, since a cluster wouldn't need to be updated right away (only the version the provider needs to offer) and the failure on the test if a version is delayed (according to the standard) would be false in this case as well.

Right. The mere existence of a cluster with outdated k8s version in it does not indicate a failure to comply with the SCS-compatibility. (For a managed k8s scenario, I would expect a provider to push customers to agree to upgrading at some point though, but we have no recommended let alone mandatory policies for this at this point.)

A test would look like this:

• We try to create a cluster with a version that should be available according to the policy.
  • If it succeeds (i.e. a working cluster is created), we are good.
  • It it fails, we are likely not.

[garloff]

2. If a new version is published by Kubernetes, I see no reason to not offer it right away. I think the versions are sufficiently tested and having a delay between the official release date and a "provided" release date just feels not right.

Well ... we currently don't offer k8s-v1.28.x officially yet, as capo does not officially support k8s-v1.28 yet.
(Yes, we provide it, it's tested and works, but we have protected it with an --allow-preview-versions flag to makr it as TechPreview.)
The up to four months is a compromise that came out of the discussion with providers. While it seems long to me, this is what everyone was willing to commit to.
For patch levels, one week seems reasonable to me -- you need to build node images and run the E2E tests at least.

[cah-hbaum]

@garloff thanks for your comments, they make this issue and standard more clear to me.
I will make a small update today in order to fix the issues coming from the comments and then make a slightly bigger update next week in order to make things more clear with regard to the overall story of this standard.

[cah-hbaum]

Update the text a bit more and rebased the whole branch in order to include the latest changes from main.

[anjastrunk]

LGTM

[garloff]

Sorry if I have missed a prior answer or discussion.
But I believe that we've had discussed mandatory requirements for version recency before. These are useful, as it allows users to know which versions she can rely on as available on all KaaS platforms that are certified as SCS-compatible on the container layer. As user, it would make my live easier, as I can then decide for validating my workload on a k8s versions that I know must be available on all SCS-compatible KaaS solutions. A should does not provide this guarantee.
So were there reasons for watering this down where I missed a discussion?
Was there a discussion that resulted in g=convincing reasons that we are really

[garloff]

Why is this not a MUST?
4 months is plenty of time to get the latest minor release built, validated, fixed (if needed, which is normally not the case) and published.

[cah-hbaum]

To answer all of the MUST questions, I had a grave misunderstanding with the way how MUST and SHOULD are used. In my mind, they were equal, but after reading the RFC document regarding MUST / SHOULD / ... I can see my mistake.
And you're right, all of the ones you pointed out should be MUST.

[garloff]

I believe we can make this a MUST as well.

[cah-hbaum]

See #334 (comment)

[garloff]

MUST?

[cah-hbaum]

See #334 (comment)

[garloff]

could be a MUST as well (and if we want to be generous, we could allow failed e2e tests to be listed in a release note rather than requiring 100% pass rate).

[cah-hbaum]

See #334 (comment)

Also I think you're right about the generosity of passed tests, but I'm not really sure how to properly display this in Release notes.

[garloff]

Again, I would make this a MUST.

[cah-hbaum]

See #334 (comment)

[garloff]

Hmm, looks duplicate to the sentence before.
And should be MUST from my PoV.

[cah-hbaum]

The first sentence was just a short introduction for the decision underneath it, but I can change that if it should be written another way.

[cah-hbaum]

Since this issue is now open for 3 months and it got one Approval by @anjastrunk and I fixed all issues opened by @garloff, I'm gonna merge this now.

[garloff]

Thanks, @cah-hbaum!