From 9a32550d1e8767c1c008d658113f57939235be6f Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 24 Jan 2024 11:22:24 +0000 Subject: [PATCH 1/6] Fix markdown issues --- runbooks/source/upgrade-eks-cluster.html.md.erb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/runbooks/source/upgrade-eks-cluster.html.md.erb b/runbooks/source/upgrade-eks-cluster.html.md.erb index 864de4b3..82714780 100644 --- a/runbooks/source/upgrade-eks-cluster.html.md.erb +++ b/runbooks/source/upgrade-eks-cluster.html.md.erb @@ -1,7 +1,7 @@ --- title: Upgrade EKS cluster weight: 53 -last_reviewed_on: 2024-01-16 +last_reviewed_on: 2024-01-24 review_in: 3 months --- @@ -155,7 +155,7 @@ Before every EKS major versions, check and upgrade if the addons versions don't After every EKS major versions, check and upgrade if the addons don't match the EKS major version the cluster you just upgraded to. -The following addons are managed through cloud-platform-terraform-eks-add-ons [module]( +The following addons are managed through cloud-platform-terraform-eks-add-ons module. [managing-kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) From a479d55237c863ccc03075c8e4a29087e257e317 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 24 Jan 2024 11:22:38 +0000 Subject: [PATCH 2/6] Add runbook for eks addons --- .../source/upgrade-eks-addons.html.md.erb | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 runbooks/source/upgrade-eks-addons.html.md.erb diff --git a/runbooks/source/upgrade-eks-addons.html.md.erb b/runbooks/source/upgrade-eks-addons.html.md.erb new file mode 100644 index 00000000..14ccbaec --- /dev/null +++ b/runbooks/source/upgrade-eks-addons.html.md.erb @@ -0,0 +1,55 @@ +--- +title: Upgrade EKS addons +weight: 53 +last_reviewed_on: 2024-01-24 +review_in: 3 months +--- + +# Upgrade EKS addons + +We have 3 addons managed through cloud-platform-terraform-eks-add-ons [module](https://github.com/ministryofjustice/cloud-platform-terraform-eks-add-ons). + +Before every EKS major versions, check and upgrade if the addons versions don't match the EKS major version the cluster is currently on. + +After every EKS major versions, check and upgrade if the addons don't match the EKS major version the cluster you just upgraded to. + +The following addons are managed through cloud-platform-terraform-eks-add-ons module. + +[managing-kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) + +[managing-coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) + +[managing-vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + +## Preparing for upgrade + +Check the changelog for each of the addons and determine if there are any breaking changes. + +Create a thread in #cloud-platform notifying the team that upgrades are starting and the pipelines will be paused. + +## Starting the upgrade + +1. Bump the version number in cloud-platform-terraform-eks-add-ons +2. Commit changes on a new branch and create a pull request +3. Request review from someone on the team +4. Merge pull request and create a new release through the Github UI +5. Bump the version number of the cloud-platform-terraform-eks-add-ons in cloud-platform-infrastructure +6. Commit changes on a new branch and create a pull request +7. Request review form someone on the team +8. Check the terraform plan in concourse and pause the following pipelines: + * bootstrap + * infrastructure-live + * infrastructure-manager + * infrastructure-live-2 +9. Create an output of the configuration of a pod before the upgrade. `kubectl -n kube-system get pod $addon -oyaml` there is also a helper [script](https://github.com/ministryofjustice/cloud-platform-infrastructure/blob/main/scripts/addons-upgrade.bash). +10. Merge the pull request +11. Unpause an infrastructure pipeline and wait for it to complete +12. While running: + * Keep an eye on pods recycling `watch -n 1 "kubectl -n kube-system get pods"` + * Keep an eye on events `watch -n 1 "kubectl -n kube-system get events"` +13. Run the reporting pipeline on the infrastructure environment +14. If everythin is green repeat steps 11-14 on each environment. + +## Finish the upgrade + +Finish up communications and close the thread. \ No newline at end of file From e991e12394a207ca53bf3d7bedb695eb93ff0d4c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 24 Jan 2024 11:23:40 +0000 Subject: [PATCH 3/6] Commit changes made by code formatters --- runbooks/source/upgrade-eks-addons.html.md.erb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runbooks/source/upgrade-eks-addons.html.md.erb b/runbooks/source/upgrade-eks-addons.html.md.erb index 14ccbaec..71d8c139 100644 --- a/runbooks/source/upgrade-eks-addons.html.md.erb +++ b/runbooks/source/upgrade-eks-addons.html.md.erb @@ -52,4 +52,4 @@ Create a thread in #cloud-platform notifying the team that upgrades are starting ## Finish the upgrade -Finish up communications and close the thread. \ No newline at end of file +Finish up communications and close the thread. From ab83764fa957b5a9a605b8c4bf9be2fb47f4af98 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 24 Jan 2024 11:45:08 +0000 Subject: [PATCH 4/6] Add eksctl info --- runbooks/source/upgrade-eks-addons.html.md.erb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/runbooks/source/upgrade-eks-addons.html.md.erb b/runbooks/source/upgrade-eks-addons.html.md.erb index 71d8c139..3720520b 100644 --- a/runbooks/source/upgrade-eks-addons.html.md.erb +++ b/runbooks/source/upgrade-eks-addons.html.md.erb @@ -21,6 +21,20 @@ The following addons are managed through cloud-platform-terraform-eks-add-ons mo [managing-vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) +## Listing available EKS upgrades + +### eksctl Install + +``` +brew install eksctl +``` + +Run the following command to get a list of supported addon version for the Kubernetes version. + +``` +eksctl utils describe-addon-versions --kubernetes-version [k8s-version] --name [addon name: kube-proxy/vpc-cni/coredns] | grep AddonVersion +``` + ## Preparing for upgrade Check the changelog for each of the addons and determine if there are any breaking changes. From 45a492dad71747e556ed4813d71b6b90209d1cc0 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 24 Jan 2024 11:52:47 +0000 Subject: [PATCH 5/6] Update runbooks/source/upgrade-eks-addons.html.md.erb Co-authored-by: Steve Williams <105657964+sj-williams@users.noreply.github.com> --- runbooks/source/upgrade-eks-addons.html.md.erb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runbooks/source/upgrade-eks-addons.html.md.erb b/runbooks/source/upgrade-eks-addons.html.md.erb index 3720520b..78a5799a 100644 --- a/runbooks/source/upgrade-eks-addons.html.md.erb +++ b/runbooks/source/upgrade-eks-addons.html.md.erb @@ -9,7 +9,7 @@ review_in: 3 months We have 3 addons managed through cloud-platform-terraform-eks-add-ons [module](https://github.com/ministryofjustice/cloud-platform-terraform-eks-add-ons). -Before every EKS major versions, check and upgrade if the addons versions don't match the EKS major version the cluster is currently on. +Before every EKS major version upgrade, check and upgrade if the addons versions don't match the EKS major version the cluster is currently on. After every EKS major versions, check and upgrade if the addons don't match the EKS major version the cluster you just upgraded to. From 5ab2fd0fff7cd0ddff1721e3a95c2cc75e843e45 Mon Sep 17 00:00:00 2001 From: Mike Bell Date: Wed, 24 Jan 2024 11:55:11 +0000 Subject: [PATCH 6/6] Fix spelling mistakes --- runbooks/source/upgrade-eks-addons.html.md.erb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/runbooks/source/upgrade-eks-addons.html.md.erb b/runbooks/source/upgrade-eks-addons.html.md.erb index 78a5799a..873461cd 100644 --- a/runbooks/source/upgrade-eks-addons.html.md.erb +++ b/runbooks/source/upgrade-eks-addons.html.md.erb @@ -11,7 +11,7 @@ We have 3 addons managed through cloud-platform-terraform-eks-add-ons [module](h Before every EKS major version upgrade, check and upgrade if the addons versions don't match the EKS major version the cluster is currently on. -After every EKS major versions, check and upgrade if the addons don't match the EKS major version the cluster you just upgraded to. +After every EKS major versions upgrade, check and upgrade if the addons don't match the EKS major version the cluster you just upgraded to. The following addons are managed through cloud-platform-terraform-eks-add-ons module. @@ -49,7 +49,7 @@ Create a thread in #cloud-platform notifying the team that upgrades are starting 4. Merge pull request and create a new release through the Github UI 5. Bump the version number of the cloud-platform-terraform-eks-add-ons in cloud-platform-infrastructure 6. Commit changes on a new branch and create a pull request -7. Request review form someone on the team +7. Request review from someone on the team 8. Check the terraform plan in concourse and pause the following pipelines: * bootstrap * infrastructure-live @@ -62,7 +62,7 @@ Create a thread in #cloud-platform notifying the team that upgrades are starting * Keep an eye on pods recycling `watch -n 1 "kubectl -n kube-system get pods"` * Keep an eye on events `watch -n 1 "kubectl -n kube-system get events"` 13. Run the reporting pipeline on the infrastructure environment -14. If everythin is green repeat steps 11-14 on each environment. +14. If everything is green repeat steps 11-14 on each environment. ## Finish the upgrade