Proto-goal: establish and document k8s maintenance procedures

[consideRatio]

Brief description

In Q2 goal #2293 we got all k8s clusters to k8s 1.24+, we also go the technical steps to take when actually making an upgrade on AWS EKS based clusters. I'd like to see us complete this work up by ensuring we establish and document k8s maintenance procedures.

Rationale & value to our mission

This is a practical goal of how to better accomplish tasks we need to get done. Without procedures on how to do things, everyone is burdened by figuring things out themselves and justifying their decisions for themselves and others.

Definition of completion and success

• A procedure on how to plan a disruptive maintenance:
  • Planning should consider individual communities on dedicated clusters and sets of communities on a shared cluster
  • The documented procedure should include an email template assuming an email is to be sent, and it should clarify clearly from where and to whom an email should be sent, and how to identify email recipients for the involved communities
  • It should involve and clarify that an announcement will be presented for users
• A procedure on how to plan a disruptive maintenance with a shared cluster's communities is documented
• A procedure on how to actually do the maintenance is documented for: GCP GKE clusters, AWS EKS clusters.
  • It is a stretch goal to have this documented for Azure AKS clusters, but not required as we only have utoronto on AKS at the moment.

Proposed leads and team

Reference and information

[consideRatio]

Mostly resolved I think.