Staged apply mode not allowed for new nodes

[hkraal]

For our clusters (which we provision using Terraform) we've pinned the kubernetes (and talos) versions as we want to control the upgrades

Luckily there is --mode=staged to steer this in the right direction but this has the side effect that this yields an invalid config for new nodes as these will be in "maintenance mode" in which the apply mode "staged" is not allowed. Setting the mode to "reboot" for new nodes is a workaround for now but this has to be set per node making GitOps a bit more hassle.

There probably is a valid reason for not allowing this mode for nodes in maintenance but I wonder what the reasons is or what's wrong with the way we're approaching this.

[smira]

There is no staged mode for maintenance, as it doesn't exist in this mode, the config is applied immediately.

The mode should be set to auto most of the time.

I'm not sure what you're trying to achieve with staged though?

[hkraal]

During the upgrade of the talos provider to 0.7.0, the Kubernetes version on our cluster (running Talos 1.7) was unexpectedly upgraded to 1.32 (note that Talos 1.7 doesn't even support this version) as we did not set the kubernetes_version in the machine config as you guessed correctly.

Bumping the Kubernetes version in the machine config will apply against all nodes were I would like have more control so that I can drain a node, update it and take it back into production. This is wanted for all changes, not just the Kubernetes version BTW. So apply mode seemed like the way to go.

[smira]

Kubernetes version shouldn't be applied node-by-node by the way.

Terraform doesn't offer a proper way, but talosctl upgrade-k8s does (incompatible with Terraform though).

Omni has a proper solution for upgrades and config management.

Anyways, --staged is a bad workaround, as Talos can update Kubernetes version on the fly without a reboot. So I think this is asking for a wrong solution to a problem.

Set the Kubernetes version to what it should be now, and it would be a no-op config change, which will be applied right away.

[hkraal]

Upgrading K8s would indeed be done via talosctl upgrade-k8s, not via Terraform.

The --mode=staged part was related as we needed to bump the talos_version as well (this seemed to be set to the initial Talos version when the cluster was created) before we could set the K8s version to 1.32. As this could also trigger reboots I used the --mode=staged to test the waters.

But that is only really needed until we have reconciled both versions to the supported matrix after which the mode can return to it's default of "auto". Somehow I ended up solving the wrong problem without looking back, thank you for setting me straight!

[smira]

You should never update talos_version, fyi, it might have some unsupported consequences. It captures the initial cluster Talos version as the machine config was generated.

Kubernetes version should be allowed anyways (if not, it's a bug).

[hkraal]

Than I suspect we do indeed have a bug as I was unable to set the kubernetes version without bumping the talos version (which was 1.6 in my case). This problem did only show when applying the Terraform config (plan was ok).

I'll try to setup a test case to confirm it and open an issue with the terraform provider repo if confirmed

[hkraal]

Confirmed the suspected bug and opened an issue with steps to reproduce; siderolabs/terraform-provider-talos#228