Document a k8s upgrade procedure for GKE

[consideRatio]

Action points

• Research and draft how to upgrade GKE clusters
  • Preliminary notes by Erik are available in callysto: maintenance plan for Wed, Mar 22, 0-4am PDT #2388 (comment).
  • Its likeley relevant to consider Decide and document on pod scaling of GKE's clusters calico-typha pods, and konnectivity-agent pods #2490, and ensure we don't get two core replicas
  • Both Erik and Yuvi has made upgrades so far, where Erik has used terraform, but I think Yuvi may have used another strategy. It could be good to have Yuvi's input about how this has been done.
• Upgrade some unused GKE k8s cluster opportunistically based on draft notes
• Document GKE upgrade procedure, equivalent of docs: add an aws k8s cluster upgrade guide #2142

Related

• docs: add an aws k8s cluster upgrade guide #2142
• Migrate existing GCP GKE based hubs from k8s 1.22+ to 1.24+ #2157 (this issue was extracted from that)
• Document a k8s upgrade procedure for AKS #3251 (sibling issue for AKS)

[consideRatio]

# useful in order to work with a different workspace in another terminal
# by default, the workspace is otherwise found in `./terraform/environment
export TF_WORKSPACE=...

[consideRatio]

It seems that terraform can perform a node surge upgrade by updating node pools version via terraform, i previously thought we couldn't, this makes us more easily able to upgrade core nodes but doesn't influence the user nodes very much as we can't evict users etc.