From 56a0504711af13969da794f366955d53180ab32a Mon Sep 17 00:00:00 2001 From: Poornima Krishnasamy Date: Fri, 23 Feb 2024 12:56:22 +0000 Subject: [PATCH 1/5] Add Kubernetes Upgrade ISSUE_TEMPLATE --- .../cloud-platform-k8s-upgrade-template.md | 177 ++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md diff --git a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md new file mode 100644 index 00000000..d4c2e406 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md @@ -0,0 +1,177 @@ +--- +name: Cloud Platform Kubernetes Upgrade story +about: Create new Cloud Platform Kubernetes Upgrade story +--- + +## Issue 1: +### Terraform AWS EKS Modules support for +Check to see if there are any issues with using with Kubernetes + +## Issue 2: +### Check for deprecated resources for +Scan to see if we have any in the production clusters live/live-2/manager. If any present, take actions to address that. + +Refer to the deprecations note on runbook: https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html#compatibility-check + +## Issue 3: +### Update vpc-cni from to the version needed for current k8s version +Check for the latest supported addon version for the current kubernetes version and update the clusters + +Production Clusters Checklist: + +- [ ] live-2 +- [ ] manager +- [ ] live + +See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) documentation for more information about addons, or find the latest versions for these EKS add-ons directly: + +- [coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) +- [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) +- [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + + +## Issue 4: +### Update kube-proxy from to the version needed for current k8s version +Check for the latest supported addon version for the current kubernetes version and update the clusters + +Production Clusters Checklist: + +- [ ] live-2 +- [ ] manager +- [ ] live + +See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) documentation for more information about addons, or find the latest versions for these EKS add-ons directly: + +- [coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) +- [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) +- [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + + +## Issue 5: +### Update core-dns from to the version needed for current k8s version +Check for the latest supported addon version for the current kubernetes version and update the clusters + +Production Clusters Checklist: + +- [ ] live-2 +- [ ] manager +- [ ] live + +See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) documentation for more information about addons, or find the latest versions for these EKS add-ons directly: + +- [coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) +- [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) +- [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + +## Issue 6: +### Test EKS on test cluster + +Follow instructions from Upgrade runbook: https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html#upgrade-eks-cluster + +## Issue 7: +### Test EKS on live-like cluster + +Follow instructions from creating a live like cluster runbook: https://runbooks.cloud-platform.service.justice.gov.uk/creating-a-live-like.html#creating-a-live-like-cluster + +## Issue 8: +### EKS: Upgrade live-2 to Kubernetes +https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html + +## Issue 9: +### EKS: Upgrade manager to Kubernetes +https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html + +## Issue 10: +### EKS: Upgrade live to Kubernetes +https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html + +## Issue 11: +### EKS: Upgrade live to Kubernetes +https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html + +## Issue 12: +### Update vpc-cni from to the version needed for the upgraded k8s version +Check for the latest supported addon version for the upgraded kubernetes version and update the clusters + +Production Clusters Checklist: + +- [ ] live-2 +- [ ] manager +- [ ] live + +See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) documentation for more information about addons, or find the latest versions for these EKS add-ons directly: + +- [coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) +- [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) +- [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + + +## Issue 13: +### Update kube-proxy from to the version needed for the upgraded k8s version +Check for the latest supported addon version for the upgraded kubernetes version and update the clusters + +Production Clusters Checklist: + +- [ ] live-2 +- [ ] manager +- [ ] live + +See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) documentation for more information about addons, or find the latest versions for these EKS add-ons directly: + +- [coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) +- [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) +- [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + + +## Issue 14: +### Update core-dns from to the version needed for the upgraded k8s version +Check for the latest supported addon version for the upgraded kubernetes version and update the clusters + +Production Clusters Checklist: + +- [ ] live-2 +- [ ] manager +- [ ] live + +See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) documentation for more information about addons, or find the latest versions for these EKS add-ons directly: + +- [coredns](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html) +- [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) +- [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) + +## Issue 15: +### Review cluster components for upgrading +https://runbooks.cloud-platform.service.justice.gov.uk/container-images.html#container-images-used-by-cluster-components + +Review the compatibility matrix for all cluster components and verify if the image is compatible with the upgraded kuebrnetes version + +## Issue 16: +### Review and upgrade kube-state-metrics for the upgraded cluster version +https://runbooks.cloud-platform.service.justice.gov.uk/container-images.html#container-images-used-by-cluster-components + +https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#compatibility-matrix + +Review the compatibility matrix of kube-state-metrics and verify if the image is compatible with the upgraded kuebrnetes version. If the version is mismatch, check if the whole kube-prometheus-chart needs updating to get the default version of kube-state-metrics from the chart-> values. Otherwise, pin the kube-state-metrics image to match the kubernetes-version + +## Issue 17: +### Upgrade cluster autoscalar for k8s version +The Cloud Platform Cluster is in k8s version . Hence upgrade the cluster-autoscalar to match the k8s version. + +https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler +https://github.com/ministryofjustice/cloud-platform-terraform-cluster-autoscaler + +## Issue 18: +### Upgrade cluster descheudler for k8s version +The Cloud Platform Cluster is in k8s version . Hence upgrade the descheduler to match the k8s version. + +https://github.com/kubernetes-sigs/descheduler?tab=readme-ov-file#%EF%B8%8F--documentation-versions-by-release +https://github.com/ministryofjustice/cloud-platform-terraform-descheduler + +## Issue 19: +### Post k8s version Cleanup +Following upgrade from EKS , there will be a number of cleanup activities that need addressing: + +Update user guide / runbook references +Update tools-image for kubectl version +Update cloud-platform-cli for kubectl version +Update concourse pipelines to use updated tools-image and cli \ No newline at end of file From 0673bdbefe3a77fd5a8d6392009d7937dbc820b7 Mon Sep 17 00:00:00 2001 From: Poornima Krishnasamy Date: Fri, 23 Feb 2024 14:52:32 +0000 Subject: [PATCH 2/5] Merge prod cluster upgrade --- .../cloud-platform-k8s-upgrade-template.md | 52 +++++++------------ 1 file changed, 18 insertions(+), 34 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md index d4c2e406..e141b518 100644 --- a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md +++ b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md @@ -4,16 +4,6 @@ about: Create new Cloud Platform Kubernetes Upgrade story --- ## Issue 1: -### Terraform AWS EKS Modules support for -Check to see if there are any issues with using with Kubernetes - -## Issue 2: -### Check for deprecated resources for -Scan to see if we have any in the production clusters live/live-2/manager. If any present, take actions to address that. - -Refer to the deprecations note on runbook: https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html#compatibility-check - -## Issue 3: ### Update vpc-cni from to the version needed for current k8s version Check for the latest supported addon version for the current kubernetes version and update the clusters @@ -30,7 +20,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 4: +## Issue 2: ### Update kube-proxy from to the version needed for current k8s version Check for the latest supported addon version for the current kubernetes version and update the clusters @@ -47,7 +37,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 5: +## Issue 3: ### Update core-dns from to the version needed for current k8s version Check for the latest supported addon version for the current kubernetes version and update the clusters @@ -63,33 +53,27 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 6: +## Issue 4: ### Test EKS on test cluster Follow instructions from Upgrade runbook: https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html#upgrade-eks-cluster -## Issue 7: +## Issue 5: ### Test EKS on live-like cluster Follow instructions from creating a live like cluster runbook: https://runbooks.cloud-platform.service.justice.gov.uk/creating-a-live-like.html#creating-a-live-like-cluster -## Issue 8: -### EKS: Upgrade live-2 to Kubernetes -https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html - -## Issue 9: -### EKS: Upgrade manager to Kubernetes +## Issue 6: +### EKS: Upgrade Production clusters to Kubernetes https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html -## Issue 10: -### EKS: Upgrade live to Kubernetes -https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html +Production Clusters Checklist: -## Issue 11: -### EKS: Upgrade live to Kubernetes -https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html +- [ ] live-2 +- [ ] manager +- [ ] live -## Issue 12: +## Issue 7: ### Update vpc-cni from to the version needed for the upgraded k8s version Check for the latest supported addon version for the upgraded kubernetes version and update the clusters @@ -106,7 +90,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 13: +## Issue 8: ### Update kube-proxy from to the version needed for the upgraded k8s version Check for the latest supported addon version for the upgraded kubernetes version and update the clusters @@ -123,7 +107,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 14: +## Issue 9: ### Update core-dns from to the version needed for the upgraded k8s version Check for the latest supported addon version for the upgraded kubernetes version and update the clusters @@ -139,13 +123,13 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 15: +## Issue 10: ### Review cluster components for upgrading https://runbooks.cloud-platform.service.justice.gov.uk/container-images.html#container-images-used-by-cluster-components Review the compatibility matrix for all cluster components and verify if the image is compatible with the upgraded kuebrnetes version -## Issue 16: +## Issue 11: ### Review and upgrade kube-state-metrics for the upgraded cluster version https://runbooks.cloud-platform.service.justice.gov.uk/container-images.html#container-images-used-by-cluster-components @@ -153,21 +137,21 @@ https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#compatibilit Review the compatibility matrix of kube-state-metrics and verify if the image is compatible with the upgraded kuebrnetes version. If the version is mismatch, check if the whole kube-prometheus-chart needs updating to get the default version of kube-state-metrics from the chart-> values. Otherwise, pin the kube-state-metrics image to match the kubernetes-version -## Issue 17: +## Issue 12: ### Upgrade cluster autoscalar for k8s version The Cloud Platform Cluster is in k8s version . Hence upgrade the cluster-autoscalar to match the k8s version. https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler https://github.com/ministryofjustice/cloud-platform-terraform-cluster-autoscaler -## Issue 18: +## Issue 13: ### Upgrade cluster descheudler for k8s version The Cloud Platform Cluster is in k8s version . Hence upgrade the descheduler to match the k8s version. https://github.com/kubernetes-sigs/descheduler?tab=readme-ov-file#%EF%B8%8F--documentation-versions-by-release https://github.com/ministryofjustice/cloud-platform-terraform-descheduler -## Issue 19: +## Issue 14: ### Post k8s version Cleanup Following upgrade from EKS , there will be a number of cleanup activities that need addressing: From d58171955e61216ad380e81b8ec0830fe4f93085 Mon Sep 17 00:00:00 2001 From: Poornima Krishnasamy Date: Wed, 28 Feb 2024 17:23:56 +0000 Subject: [PATCH 3/5] Add script to generate issues from upgrade epic ticket --- cmd/create-upgrade-issues/go.mod | 17 ++++++ cmd/create-upgrade-issues/go.sum | 35 ++++++++++++ cmd/create-upgrade-issues/main.go | 88 +++++++++++++++++++++++++++++++ 3 files changed, 140 insertions(+) create mode 100644 cmd/create-upgrade-issues/go.mod create mode 100644 cmd/create-upgrade-issues/go.sum create mode 100644 cmd/create-upgrade-issues/main.go diff --git a/cmd/create-upgrade-issues/go.mod b/cmd/create-upgrade-issues/go.mod new file mode 100644 index 00000000..a8574610 --- /dev/null +++ b/cmd/create-upgrade-issues/go.mod @@ -0,0 +1,17 @@ +module ministryofjustice/cloud-platform/cmd/create-upgrade-issues + +go 1.21.7 + +require ( + github.com/google/go-github/v35 v35.3.0 + golang.org/x/oauth2 v0.17.0 +) + +require ( + github.com/golang/protobuf v1.5.3 // indirect + github.com/google/go-querystring v1.0.0 // indirect + golang.org/x/crypto v0.19.0 // indirect + golang.org/x/net v0.21.0 // indirect + google.golang.org/appengine v1.6.7 // indirect + google.golang.org/protobuf v1.31.0 // indirect +) diff --git a/cmd/create-upgrade-issues/go.sum b/cmd/create-upgrade-issues/go.sum new file mode 100644 index 00000000..da544928 --- /dev/null +++ b/cmd/create-upgrade-issues/go.sum @@ -0,0 +1,35 @@ +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= +github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= +github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= +github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-github/v35 v35.3.0 h1:fU+WBzuukn0VssbayTT+Zo3/ESKX9JYWjbZTLOTEyho= +github.com/google/go-github/v35 v35.3.0/go.mod h1:yWB7uCcVWaUbUP74Aq3whuMySRMatyRmq5U9FTNlbio= +github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk= +github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.19.0 h1:ENy+Az/9Y1vSrlrvBSyna3PITt4tiZLf7sgCjZBX7Wo= +golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= +golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4= +golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.17.0 h1:6m3ZPmLEFdVxKKWnKq4VqZ60gutO35zm+zrAHVmHyDQ= +golang.org/x/oauth2 v0.17.0/go.mod h1:OzPDGQiuQMguemayvdylqddI7qcD9lnSDb+1FiwQ5HA= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= +google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= +google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8= +google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= diff --git a/cmd/create-upgrade-issues/main.go b/cmd/create-upgrade-issues/main.go new file mode 100644 index 00000000..9e7b53c1 --- /dev/null +++ b/cmd/create-upgrade-issues/main.go @@ -0,0 +1,88 @@ +package main + +import ( + "context" + "flag" + "fmt" + "os" + "strings" + + "github.com/google/go-github/v35/github" + "golang.org/x/oauth2" +) + +// GitHubAccess contains GitHub access information +type GitHubAccess struct { + AccessToken string + RepoOwner string + RepoName string + IssueNumber int +} + +// ParseIssue parses the given issue and returns a slice of issues +func ParseIssue(issue string) []string { + var issues []string + sections := strings.Split(issue, "## Issue") + + for _, section := range sections { + if section != "" { + lines := strings.Split(strings.TrimSpace(section), "\n") + title := strings.TrimSpace(lines[0]) + body := strings.TrimSpace(strings.Join(lines[1:], "\n")) + issues = append(issues, fmt.Sprintf("## %s\n%s", title, body)) + } + } + return issues +} + +// CreateIssue creates a GitHub issue +func CreateIssue(client *github.Client, ghAccess GitHubAccess, issue string, epicIssueNumber int) error { + issueParts := strings.SplitN(issue, "\n", 3) + title := strings.TrimSpace(strings.TrimPrefix(issueParts[1], "###")) + body := strings.TrimSpace(issueParts[2]) + + // Append the body with the epic issue number + body = fmt.Sprintf("%s\n\n Related to: #%d", body, epicIssueNumber) + + issueRequest := &github.IssueRequest{ + Title: &title, + Body: &body, + } + _, _, err := client.Issues.Create(context.Background(), ghAccess.RepoOwner, ghAccess.RepoName, issueRequest) + return err +} + +func main() { + + var ghAccess GitHubAccess + ghAccess.AccessToken = os.Getenv("GITHUB_TOKEN") + flag.StringVar(&ghAccess.RepoOwner, "owner", "ministryofjustice", "the repository to create issues") + flag.StringVar(&ghAccess.RepoName, "repo", "cloud-platform", "the repository to create issues") + flag.IntVar(&ghAccess.IssueNumber, "issue", 0, "the issue number to create issues") + + flag.Parse() + + ctx := context.Background() + ts := oauth2.StaticTokenSource( + &oauth2.Token{AccessToken: ghAccess.AccessToken}, + ) + tc := oauth2.NewClient(ctx, ts) + client := github.NewClient(tc) + + // Fetch the issue contents from GitHub + issueContent, _, err := client.Issues.Get(ctx, ghAccess.RepoOwner, ghAccess.RepoName, ghAccess.IssueNumber) + if err != nil { + fmt.Println("Error fetching issue template:", err) + os.Exit(1) + } + + issues := ParseIssue(*issueContent.Body) + for _, issue := range issues { + //fmt.Println("Creating issue:", issue) + err := CreateIssue(client, ghAccess, issue, ghAccess.IssueNumber) + if err != nil { + fmt.Println("Error creating issue:", err) + os.Exit(1) + } + } +} From a0cb4f2baa98c1323901d5f31a1c5c5eff6db896 Mon Sep 17 00:00:00 2001 From: Poornima Krishnasamy Date: Thu, 29 Feb 2024 17:32:43 +0000 Subject: [PATCH 4/5] Update about info and add label --- .github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md index e141b518..0e5d05ce 100644 --- a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md +++ b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md @@ -1,6 +1,7 @@ --- name: Cloud Platform Kubernetes Upgrade story -about: Create new Cloud Platform Kubernetes Upgrade story +about: This template is for Cloud Platform team to create an epic ticket for kubernetes upgrades +label: 'EPIC' --- ## Issue 1: From 98593a012e23922e289427956d218e4ba88b7d16 Mon Sep 17 00:00:00 2001 From: Poornima Krishnasamy Date: Fri, 1 Mar 2024 16:42:02 +0000 Subject: [PATCH 5/5] Add planning and updating runbook steps --- .../cloud-platform-k8s-upgrade-template.md | 59 ++++++++++++++----- 1 file changed, 45 insertions(+), 14 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md index 0e5d05ce..7a20a6bd 100644 --- a/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md +++ b/.github/ISSUE_TEMPLATE/cloud-platform-k8s-upgrade-template.md @@ -5,6 +5,26 @@ label: 'EPIC' --- ## Issue 1: +### Planning upgrade to EKS + +Go through the release notes of EKS 1.27 and create a plan to upgrade our clusters + +Things to consider: + +Review changelog & release notes +EKS Module support at current version? +Are there any API deprecations & removals? (Check insights) +Are there new components being added? +What changes are being introduced to current components? +Are there changes to core infra of the CP required? i.e. Are all our current components compatible with ? +Are there changes users need to make? +Do we need to expand any of our smoke/integration testing? +Create additional tickets needed for any findings specific to this upgrade + +Cluster upgrade Runbook: +https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html + +## Issue 2: ### Update vpc-cni from to the version needed for current k8s version Check for the latest supported addon version for the current kubernetes version and update the clusters @@ -21,7 +41,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 2: +## Issue 3: ### Update kube-proxy from to the version needed for current k8s version Check for the latest supported addon version for the current kubernetes version and update the clusters @@ -38,7 +58,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 3: +## Issue 4: ### Update core-dns from to the version needed for current k8s version Check for the latest supported addon version for the current kubernetes version and update the clusters @@ -54,17 +74,17 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 4: +## Issue 5: ### Test EKS on test cluster Follow instructions from Upgrade runbook: https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html#upgrade-eks-cluster -## Issue 5: +## Issue 6: ### Test EKS on live-like cluster Follow instructions from creating a live like cluster runbook: https://runbooks.cloud-platform.service.justice.gov.uk/creating-a-live-like.html#creating-a-live-like-cluster -## Issue 6: +## Issue 7: ### EKS: Upgrade Production clusters to Kubernetes https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html @@ -74,7 +94,7 @@ Production Clusters Checklist: - [ ] manager - [ ] live -## Issue 7: +## Issue 8: ### Update vpc-cni from to the version needed for the upgraded k8s version Check for the latest supported addon version for the upgraded kubernetes version and update the clusters @@ -91,7 +111,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 8: +## Issue 9: ### Update kube-proxy from to the version needed for the upgraded k8s version Check for the latest supported addon version for the upgraded kubernetes version and update the clusters @@ -108,7 +128,7 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 9: +## Issue 10: ### Update core-dns from to the version needed for the upgraded k8s version Check for the latest supported addon version for the upgraded kubernetes version and update the clusters @@ -124,13 +144,13 @@ See the [Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/ek - [kube-proxy](https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html) - [vpc-cni](https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html) -## Issue 10: +## Issue 11: ### Review cluster components for upgrading https://runbooks.cloud-platform.service.justice.gov.uk/container-images.html#container-images-used-by-cluster-components Review the compatibility matrix for all cluster components and verify if the image is compatible with the upgraded kuebrnetes version -## Issue 11: +## Issue 12: ### Review and upgrade kube-state-metrics for the upgraded cluster version https://runbooks.cloud-platform.service.justice.gov.uk/container-images.html#container-images-used-by-cluster-components @@ -138,25 +158,36 @@ https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#compatibilit Review the compatibility matrix of kube-state-metrics and verify if the image is compatible with the upgraded kuebrnetes version. If the version is mismatch, check if the whole kube-prometheus-chart needs updating to get the default version of kube-state-metrics from the chart-> values. Otherwise, pin the kube-state-metrics image to match the kubernetes-version -## Issue 12: +## Issue 13: ### Upgrade cluster autoscalar for k8s version The Cloud Platform Cluster is in k8s version . Hence upgrade the cluster-autoscalar to match the k8s version. https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler https://github.com/ministryofjustice/cloud-platform-terraform-cluster-autoscaler -## Issue 13: +## Issue 14: ### Upgrade cluster descheudler for k8s version The Cloud Platform Cluster is in k8s version . Hence upgrade the descheduler to match the k8s version. https://github.com/kubernetes-sigs/descheduler?tab=readme-ov-file#%EF%B8%8F--documentation-versions-by-release https://github.com/ministryofjustice/cloud-platform-terraform-descheduler -## Issue 14: +## Issue 15: ### Post k8s version Cleanup Following upgrade from EKS , there will be a number of cleanup activities that need addressing: Update user guide / runbook references Update tools-image for kubectl version Update cloud-platform-cli for kubectl version -Update concourse pipelines to use updated tools-image and cli \ No newline at end of file +Update concourse pipelines to use updated tools-image and cli + +## Issue 16: +### Update Upgrade runbook and Issue template: + +Update the runbook for +- any changes needed in the steps to perform the upgrade +- any lessons learnt that could be useful for next upgrade +- any changes to the upgrade issue template + +Cluster upgrade Runbook: +https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html