From 3795372b11e7a96ad76e24d904caf8b769977cb5 Mon Sep 17 00:00:00 2001 From: igooch Date: Fri, 13 Dec 2024 10:12:55 -0800 Subject: [PATCH] In place upgrades version update instructions (#4064) * Adds instructions for upgrading Kubernetes version of Upgrade test clusters * Updates .md file formatting * Adds sed command to update the Golang version for upgrade test Dockerfiles * Adds pre- and post- Agones release instructions for updating upgrade test versions --- .../ISSUE_TEMPLATE/golang_version_upgrade.md | 20 +- .github/ISSUE_TEMPLATE/kubernetes_update.md | 172 ++++++++++++------ docs/governance/templates/release_issue.md | 11 +- test/sdk/go/Dockerfile | 1 - 4 files changed, 138 insertions(+), 66 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/golang_version_upgrade.md b/.github/ISSUE_TEMPLATE/golang_version_upgrade.md index 42326bb316..95b020c46c 100644 --- a/.github/ISSUE_TEMPLATE/golang_version_upgrade.md +++ b/.github/ISSUE_TEMPLATE/golang_version_upgrade.md @@ -9,19 +9,23 @@ assignees: '' Steps to upgrade Golang version: -- [ ] Update `go.mod` and `go.sum`. At the root of the directory, run: +- [ ] Update `go.mod` and `go.sum`. At the root (agones) directory, run: - [ ] `find . -name 'go.mod' -not -path '*/\.*' -execdir go mod edit -go= \;` - [ ] `find . -name 'go.mod' -not -path '*/\.*' -execdir go mod tidy \;` -- [ ] Update the Dockerfiles for `build` directory. - - - [ ] At the root of the directory, run: `find build -type f \( -not -path '*/\.*' -and -not -path 'build/tmp/*' \) -exec sed -i 's/GO_VERSION=[0-9]\+\.[0-9]\+\.[0-9]\+/GO_VERSION=/g' {} \;` +- [ ] Update the Dockerfiles for `build` directory. + - [ ] At the root directory, run: `find build -type f \( -not -path '*/\.*' -and -not -path 'build/tmp/*' \) -exec sed -i 's/GO_VERSION=[0-9]\+\.[0-9]\+\.[0-9]\+/GO_VERSION=/g' {} \;` - [ ] Update the `golang` version for file `build/agones-bot/Dockerfile` to - -- [ ] Update the Dockerfiles for `examples` directory. At the root of the directory, run: + +- [ ] Update the Dockerfiles for `test` directory. + - [ ] At the root directory, run: `find test -type f -exec sed -i 's/golang:[0-9]\+\.[0-9]\+\.[0-9]\+/golang:/g' {} \;` + - [ ] At the root directory, run: `find test -type f -exec sed -i 's/go [0-9]\+\.[0-9]\+\.[0-9]\+/go /g' {} \;` + - [ ] At the root directory, run: `find test -type f -name 'go.mod' -execdir go mod tidy \;` + +- [ ] Update the Dockerfiles for `examples` directory. At the root directory, run: - [ ] `find examples -name Dockerfile -exec sed -i 's/golang:[0-9]\+\.[0-9]\+-alpine/golang:-alpine/g' {} \;` - [ ] `find examples \( -name Dockerfile -o -name Dockerfile.windows \) -exec sed -i 's/golang:[0-9]\+\.[0-9]\+\.[0-9]\+/golang:/g' {} \;` - + - [ ] Update the example images tag. At `build` directory, run: - [ ] `make bump-image IMAGENAME=allocation-endpoint-proxy VERSION=` - [ ] `make bump-image IMAGENAME=autoscaler-webhook VERSION=` @@ -44,4 +48,4 @@ Steps to upgrade Golang version: - [ ] Create a PR for the above changes and send for review -- [ ] Merge the above PR after it is approved \ No newline at end of file +- [ ] Merge the above PR after it is approved diff --git a/.github/ISSUE_TEMPLATE/kubernetes_update.md b/.github/ISSUE_TEMPLATE/kubernetes_update.md index 020ab1f53b..597d786957 100644 --- a/.github/ISSUE_TEMPLATE/kubernetes_update.md +++ b/.github/ISSUE_TEMPLATE/kubernetes_update.md @@ -7,68 +7,128 @@ assignees: '' --- -Agones supports and is tested against 3 releases of Kubernetes, targeting the newest version as being the [default version in the GKE Rapid channel](https://cloud.google.com/kubernetes-engine/docs/release-notes#current_versions). The vendored version of client-go will be aligned with the middle of the three supported Kubernetes versions ({version_2}). All the example clusters will use the middle of the three supported Kubernetes versions ({version_2}). +Agones supports and is tested against 3 releases of Kubernetes, targeting the newest version as +being the [default version in the GKE Rapid channel](https://cloud.google.com/kubernetes-engine/docs/release-notes#current_versions). +The vendored version of client-go will be aligned with the middle of the three supported Kubernetes +versions ({version_2}). All the example clusters will use the middle of the three supported +Kubernetes versions ({version_2}). List of items to do for upgrading to {version_1} {version_2} {version_3} - [ ] Update the cluster version of terraform submodules in `install/terraform/modules` - - [ ] Update Kubernetes version of GKE cluster (both `gke` and `gke-autopilot`) to {version_2} - - [ ] Update Kubernetes version of AKS to the newest supported version in {version_1} {version_2} {version_3} - - [ ] Update Kubernetes version of EKS to the newest supported version in {version_1} {version_2} {version_3} -- [ ] Update kubectl in dev tooling to {version_2}, the latest patch version can be found [here](https://kubernetes.io/releases/) - - [ ] Update kubectl in `build/build-image/Dockerfile` - - [ ] Update kubectl in `build/e2e-image/Dockerfile` + - [ ] Update Kubernetes version of GKE cluster (both `gke` and `gke-autopilot`) to {version_2} + - [ ] Update Kubernetes version of AKS to the newest supported version in {version_1} {version_2} + {version_3} + - [ ] Update Kubernetes version of EKS to the newest supported version in {version_1} {version_2} + {version_3} +- [ ] Update kubectl in dev tooling to {version_2}, the latest patch version can be found + [here](https://kubernetes.io/releases/) + - [ ] Update kubectl in `build/build-image/Dockerfile` + - [ ] Update kubectl in `build/e2e-image/Dockerfile` + - [ ] Update kubectl in `test/upgrade/Dockerfile` - [ ] Update the Kubernetes version of the below test clusters to {version_2} - - [ ] Minikube in `build/includes/minikube.mk` (Get the patch version [here](https://kubernetes.io/releases/) since minikube supports the latest Kubernetes release) - - [ ] Kind in `build/includes/kind.mk` (Confirm {version_2} is supported and get the patch version [here](https://github.com/kubernetes-sigs/kind/releases)) -- [ ] Update the k8s image used in the helm [pre-delete-hook](https://github.com/googleforgames/agones/blob/main/install/helm/agones/templates/hooks/pre_delete_hook.yaml) to {version_2} (Get the patch version [here](https://hub.docker.com/r/bitnami/kubectl)) -- [ ] Update client-go in `go.mod` to {version_2} by running `go get k8s.io/client-go@{CORRESPONDING_VERSION}` and `go get k8s.io/apiextensions-apiserver@{CORRESPONDING_VERSION}`, then re-run `go mod tidy` and `go mod vendor` + - [ ] Minikube in `build/includes/minikube.mk` (Get the patch version [here](https://kubernetes.io/releases/) + since minikube supports the latest Kubernetes release) + - [ ] Kind in `build/includes/kind.mk` (Confirm {version_2} is supported and get the patch version + [here](https://github.com/kubernetes-sigs/kind/releases)) +- [ ] Update the k8s image used in the helm [pre-delete-hook](https://github.com/googleforgames/agones/blob/main/install/helm/agones/templates/hooks/pre_delete_hook.yaml) + to {version_2} (Get the patch version [here](https://hub.docker.com/r/bitnami/kubectl)) +- [ ] Update client-go in `go.mod` to {version_2} by running `go get k8s.io/client-go@{CORRESPONDING_VERSION}` + and `go get k8s.io/apiextensions-apiserver@{CORRESPONDING_VERSION}`, then re-run `go mod tidy` + and `go mod vendor` - [ ] Update CRD API reference to {version_2} - - [ ] Update links to k8s documentation in `site/assets/templates/crd-doc-config.json` - - [ ] Regenerate crd api reference docs - `make gen-api-docs` - - [ ] Regenerate crd client libraries and generated code - `make gen-crd-code` + - [ ] Update links to k8s documentation in `site/assets/templates/crd-doc-config.json` + - [ ] Regenerate crd api reference docs - `make gen-api-docs` + - [ ] Regenerate crd client libraries and generated code - `make gen-crd-code` - [ ] Regenerate Kubernetes resource includes (e.g. ObjectMeta, PodTemplateSpec) - - [ ] Start a cluster with `make gcloud-test-cluster` (this cluster will use Kubernetes {version_2}), uninstall agones using `helm uninstall agones -n agones-system`, and then run `make gen-embedded-openapi` and `make gen-install` -- [ ] Update documentation for creating clusters and k8s API references to align with the above clusters versions and the k8s API version - - [ ] `site/config.toml` - - [ ] `dev_supported_k8s`, which are {version_1} {version_2} {version_3} - - [ ] `dev_k8s_api_version`, which is {version_2} - - [ ] `dev_gke_example_cluster_version`, which is {version_2} - - [ ] `dev_aks_example_cluster_version`, which is the newest AKS supported version in {version_1} {version_2} {version_3} - - [ ] `dev_eks_example_cluster_version`, which is the newest EKS supported version in {version_1} {version_2} {version_3} - - [ ] `dev_minikube_example_cluster_version`, which is {version_2} with the supported patch version + - [ ] Start a cluster with `make gcloud-test-cluster` (this cluster will use Kubernetes + {version_2}), uninstall agones using `helm uninstall agones -n agones-system`, and then run + `make gen-embedded-openapi` and `make gen-install` +- [ ] Update documentation for creating clusters and k8s API references to align with the above + clusters versions and the k8s API version + - [ ] `site/config.toml` + - [ ] `dev_supported_k8s`, which are {version_1} {version_2} {version_3} + - [ ] `dev_k8s_api_version`, which is {version_2} + - [ ] `dev_gke_example_cluster_version`, which is {version_2} + - [ ] `dev_aks_example_cluster_version`, which is the newest AKS supported version in + {version_1} {version_2} {version_3} + - [ ] `dev_eks_example_cluster_version`, which is the newest EKS supported version in + {version_1} {version_2} {version_3} + - [ ] `dev_minikube_example_cluster_version`, which is {version_2} with the supported patch version - [ ] If client-go pulled in a new version of gRPC, then also - - [ ] Update the `grpc_release_tag` in the SDK [base image grpc version](https://github.com/googleforgames/agones/blob/main/build/includes/sdk.mk). - - [ ] Update the gRPC version number in C++ gRPC Dependency documentation [here](https://github.com/googleforgames/agones/blob/main/site/content/en/docs/Guides/Client%20SDKs/cpp.md). - - [ ] Update the gRPC version - ([Dockerfile](https://github.com/googleforgames/agones/blob/main/examples/cpp-simple/Dockerfile)) - - [ ] Update the C++ `cpp-simple` image. - - [ ] Update the `cpp-simple` example images tag. At `build` directory, run: - - [ ] `make bump-image IMAGENAME=cpp-simple-server VERSION=` - - [ ] Run the following to generate and push the new `cpp-simple` example images: - - [ ] In `examples/cpp-simple`, run: `make cloud-build` - - [ ] Regenerate all client sdks: [make gen-all-sdk-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-all-sdk-grpc) - This can take 20 minutes or so, as the above changes force a rebuild. Plan your day accordingly πŸ˜ƒ. - - [ ] Regenerate allocated API endpoints: [make gen-allocation-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-allocation-grpc) -- [ ] Confirm the update works as expected by running e2e tests - - [ ] Add the new supported Kubernetes versions to the e2e clusters creation - - [ ] In `build/terraform/e2e/module.tf`, add the new supported version to the map `kubernetes_versions` following the instructions in the comment. We maintain sufficient quota for `CPUs` and `In-use Addresses` in `us-east1`, `us-west1`, `europe-west1`, and `asia-east1`, but we only have capacity for one version per region - follow the instructions to rotate the regions through new versions. (Keep the clusters on `RAPID` - we dogfood early versions in CI.) - - [ ] Recreate clusters with new scripts: `cd build; make GCP_PROJECT=agones-images gcloud-e2e-test-cluster` - - [ ] Update the Cloud Build configuration to run e2e test on the new created clusters, and disable the e2e test on the cluster with the oldest supported K8s version - - [ ] Update the `versionsAndRegions` variable to add the new supported version and remove the oldest supported K8s version in `cloudbuild.yaml` `submit-e2e-test-cloud-build` step - - [ ] Update the `featureWithGateByVersion` variable to add the new supported version and remove the oldest supported K8s version in `cloudbuild.yaml` `submit-e2e-test-cloud-build` step - - [ ] Run `make lint` for code quality check. - - [ ] Submit a PR to trigger the e2e tests and verfiy they all pass - - [ ] After the PR that includes the above Cloud Build configuration change has been merged and all the existing pending PRs in the Cloud Build queue have picked up the new configuration, submit a separate PR to update the e2e clusters terraform module to remove the e2e cluster with the oldest supported K8s version. - - [ ] In `build/terraform/e2e/module.tf`, continue following the instructions in the comment to update the `kubernetes_versions` map. - - [ ] Destroy the old clusters with new scripts: `cd build; make GCP_PROJECT=agones-images gcloud-e2e-test-cluster` + - [ ] Update the `grpc_release_tag` in the SDK [base image grpc version](https://github.com/googleforgames/agones/blob/main/build/includes/sdk.mk). + - [ ] Update the gRPC version number in C++ gRPC Dependency documentation [here](https://github.com/googleforgames/agones/blob/main/site/content/en/docs/Guides/Client%20SDKs/cpp.md). + - [ ] Update the gRPC version ([Dockerfile](https://github.com/googleforgames/agones/blob/main/examples/cpp-simple/Dockerfile)) + - [ ] Update the C++ `cpp-simple` image. + - [ ] Update the `cpp-simple` example images tag. At `build` directory, run: + - [ ] `make bump-image IMAGENAME=cpp-simple-server VERSION=` + - [ ] Run the following to generate and push the new `cpp-simple` example images: + - [ ] In `examples/cpp-simple`, run: `make cloud-build` + - [ ] Regenerate all client sdks: [make gen-all-sdk-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-all-sdk-grpc) + This can take 20 minutes or so, as the above changes force a rebuild. Plan your day accordingly πŸ˜ƒ. + - [ ] Regenerate allocated API endpoints: [make gen-allocation-grpc](https://github.com/googleforgames/agones/blob/main/build/README.md#make-gen-allocation-grpc) +- [ ] Confirm the update works as expected by running e2e and upgrade tests + - [ ] Add the new supported Kubernetes versions to the e2e clusters creation + - [ ] In `build/terraform/e2e/module.tf`, add the new supported version to the map + `kubernetes_versions` following the instructions in the comment. We maintain sufficient + quota for `CPUs` and `In-use Addresses` in `us-east1`, `us-west1`, `europe-west1`, and + `asia-east1`, but we only have capacity for one version per region - follow the + instructions to rotate the regions through new versions. (Keep the clusters on `RAPID` - + we dogfood early versions in CI.) + - [ ] Recreate clusters with new scripts: `cd build; make GCP_PROJECT=agones-images gcloud-e2e-test-cluster` + - [ ] Add the new supported Kubernetes versions to the upgrade clusters creation + - [ ] In `build/terraform/upgrade/module.tf`, add the new supported version to the map + `kubernetes_versions` following the instructions in the comment. Follow the instructions + to rotate the regions through new versions. (Keep the clusters on `RAPID`.) Note that the + upgrade clusters do not use the exact same regions as the e2e tests. + - [ ] Recreate the clusters with terraform: + ``` + cd build; make shell; cd build/terraform/upgrade + terraform init -backend-config="bucket=agones-images-upgrade-infra-bucket-tfstate" -backend-config="prefix=terraform/state" + terraform apply -var project="agones-images" + ``` + - [ ] Update the Cloud Build configuration to run e2e and ugprade tests on the new created clusters, + and disable the e2e and ugprade tests on the clusters with the oldest supported K8s version + - [ ] Update the `versionsAndRegions` variable to add the new supported version and remove the + oldest supported K8s version in `cloudbuild.yaml` `submit-e2e-test-cloud-build` step + - [ ] Update the `featureWithGateByVersion` variable to add the new supported version and remove + the oldest supported K8s version in `cloudbuild.yaml` `submit-e2e-test-cloud-build` step + - [ ] Update the `versionsAndRegions` variable to add the new supported version and remove the + oldest supported K8s version in `cloudbuild.yaml` `submit-upgrade-test-cloud-build` step + - [ ] Update the `test/upgrade/versionMap.yaml` `k8sToAgonesVersions` to add the new Kubernetes + version. For example: + ``` + ], + "1.32": [ + "Dev" + ] + ``` + - [ ] Run `make lint` for code quality check. + - [ ] Submit a PR to trigger the e2e and upgrade tests and verfiy they all pass + - [ ] After the PR that includes the above Cloud Build configuration change has been merged and + all the existing pending PRs in the Cloud Build queue have picked up the new configuration, + submit a separate PR to update the e2e and upgrade clusters terraform modules to remove the + e2e and upgrade clusters with the oldest supported K8s version. + - [ ] In `build/terraform/e2e/module.tf`, continue following the instructions in the comment to + update the `kubernetes_versions` map. + - [ ] Destroy the old clusters with new scripts: `cd build; make GCP_PROJECT=agones-images gcloud-e2e-test-cluster` + - [ ] In `build/terraform/upgrade/module.tf`, continue following the instructions in the comment + to update the `kubernetes_versions` map. + - [ ] Run the same terraform command as in the previous step: + ``` + cd build; make shell; cd build/terraform/upgrade + terraform init -backend-config="bucket=agones-images-upgrade-infra-bucket-tfstate" -backend-config="prefix=terraform/state" + terraform apply -var project="agones-images" + ``` - [ ] Recreate the performance test cluster, and config the performance test to run on the new cluster - - [ ] In `build/terraform/performance/module.tf`, update the `kubernetes_versions` to {version_2} and make sure the region is always set to `us-central1`. - - [ ] Recreate the cluster with the new script: - ``` - cd build; make shell; cd build/terraform/performance - terraform init -backend-config="bucket=agones-images-performance-infra-bucket-tfstate" -backend-config="prefix=terraform/state" - terraform apply -var project="agones-images" - ``` - - [ ] Update the `_TEST_CLUSTER_NAME` in `ci/perf-test-cloudbuild.yaml` to the name of the new created performance test cluster, and send a PR with the changes. -- \ No newline at end of file + - [ ] In `build/terraform/performance/module.tf`, update the `kubernetes_versions` to {version_2} + and make sure the region is always set to `us-central1`. + - [ ] Recreate the cluster with the new script: + ``` + cd build; make shell; cd build/terraform/performance + terraform init -backend-config="bucket=agones-images-performance-infra-bucket-tfstate" -backend-config="prefix=terraform/state" + terraform apply -var project="agones-images" + ``` + - [ ] Update the `_TEST_CLUSTER_NAME` in `ci/perf-test-cloudbuild.yaml` to the name of the newly + created performance test cluster, and send a PR with the changes. +- diff --git a/docs/governance/templates/release_issue.md b/docs/governance/templates/release_issue.md index ca1c86b855..bb96d9f816 100644 --- a/docs/governance/templates/release_issue.md +++ b/docs/governance/templates/release_issue.md @@ -40,6 +40,8 @@ and copy it into a release issue. Fill in relevant values, found inside {} - [ ] config.toml updates: - [ ] Run `make site-config-update-version` to update the release version and sync data between dev and prod. - [ ] Update documentation with updated example images tags. +- [ ] Ensure that the `alphaGates` and `betaGates` for `"Dev"` in `test/upgrade/versionMap.yaml` + match the Alpha features and Beta features in `pkg/util/runtime/features.go`. - [ ] Create PR with these changes, and merge them with an approval. - [ ] Run `git remote update && git checkout main && git reset --hard upstream/main` to ensure your code is in line with upstream (unless this is a hotfix, then do the same, but for the release branch) @@ -56,12 +58,19 @@ and copy it into a release issue. Fill in relevant values, found inside {} - [ ] Copy any review changes from the release blog post into the draft GitHub release. - [ ] Publish the draft GitHub Release. - [ ] Run `make release-branch` to create a release branch and run `gcloud config configurations activate ` to switch Agones development tooling off of the `agones-images` project. -- [ ] Email mailing lists with the release details (copy-paste the release blog post). Refer to the [Internal Mailing list posting guide][Internal Mailing list posting guide] for details. +- [ ] Email mailing lists with the release details (copy-paste the release blog post). Refer to the [Internal Mailing list posting guide][Internal Mailing list posting guide] for details. - [ ] Paste the announcement blog post to the #users Slack group. - [ ] Post to the [agonesdev](https://twitter.com/agonesdev) Twitter account. - [ ] Run `git checkout main`. - [ ] Run `make sdk-publish-rust`. This command executes `cargo login` for authentication, performs a dry-run publish, and if that succeeds, does the actual publish. Will need [crate's API TOKEN](https://crates.io/settings/tokens) from your crate's account. - [ ] Run `make sdk-update-version release_stage=after version={version}` file. This command will update the SDKs and install directories files with `{version}+1-dev` and will also set `{version}+1` in `build/Makefile`. +- [ ] In `test/sdk/go/Makefile` change `release_version` to `{version}`. + - [ ] Run `make shell` and run `gcloud config configurations activate agones-images`. + - [ ] Within the shell `cd` to the `test/sdk/go/` directory and run `make cloud-build`. +- [ ] In `test/upgrade/Makefile` change `base_version` to `{version}`. +- [ ] In `test/upgrade/versionMap.yaml` change `ReleaseVersion` to `{version}`. + - [ ] In `test/upgrade/versionMap.yaml` copy and paste `β€œDev”` in `agonesVersionFeatureGates` map. + - [ ] Change one of the `β€œDev”` in `agonesVersionFeatureGates` to `{version}`. - [ ] Create PR with these changes, and merge them with approval - [ ] Close this issue. _Congratulations!_ - the release is now complete! :tada: :clap: :smile: :+1: diff --git a/test/sdk/go/Dockerfile b/test/sdk/go/Dockerfile index fe0ca478cd..e0059fcdd8 100644 --- a/test/sdk/go/Dockerfile +++ b/test/sdk/go/Dockerfile @@ -21,7 +21,6 @@ COPY *.go ./ # Note: because this installs the most recently released version of Agones, this will need to be # built and pushed post-release. -# TODO: Add to post-release check list. RUN go mod init agones.dev/agones/test/sdk/go/sdk-client-test RUN go mod tidy RUN go mod download