Decide on a cluster upgrade policy

[yuvipanda]

Background

We already have 6 clusters across 3 cloud providers, and this number is only going to increase. Currently they are on different k8s versions based on the cloud provider, even though they are mostly deployed the same way. This will eventually bite us.

I propose we adopt the following rules:

1. We try to keep all our clusters to the same version that's the default on new GKE clusters. GKE is often the last to adopt a kubernetes master version, so this gives us time to adapt.
2. Do upgrades on a 2 month cycle. So every two months, we find the current GKE default version, and upgrade all our clusters to it.
3. Upgrades take place over the course of a week
4. We find some way to communicate this 'upgrade window' to our users. It should be as unobtrusive to them as possible. For centrally managed hubs (pilot-hubs and cloudbank right now), we need to just provide notice.
5. Do both master and node upgrades as part of the same window.

Expected timeline

This issue is only about discussing and agreeing on a timeline, and not actually doing anything. So it's complete once we document it.

[damianavila]

I agree with the proposal.
Maybe one upgrade per quarter instead of every 2 month?
Once we have agreed in all the points, I think it would be useful to have some status page showing this upgrade event (and maintenance window) in addition to some email notice...

[yuvipanda]

Quarter makes sense, since kubernetes itself is on a 3 month upgrade cycle.

Once we have agreed in all the points, I think it would be useful to have some status page showing this upgrade event (and maintenance window) in addition to some email notice...

What do you think of using statuspage.io for this?

[yuvipanda]

Should we set up calendar entries? How do we execute this?

[damianavila]

Quarter makes sense, since kubernetes itself is on a 3 month upgrade cycle.

Super!

What do you think of using statuspage.io for this?

+1

Should we set up calendar entries? How do we execute this?

Calendar entries seem a reasonable way... and we create a ticket for each of these operations to keep track of the process.

[choldgraf]

I opened up #608 to track the status page, but I think that we can tackle this issue (having a policy for upgrading clusters) without blocking on that one. It seems more important that we have a policy and follow it, to make sure that we standardize our infrastructure and don't have large update jumps.

[sgibson91]

cross-reffing 2i2c-org/team-compass#423 that discusses point 4

[consideRatio]

For reference, this issue provides an overview of k8s versions in clusters currently: #2057