Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: 🐛 broken page #5092

Merged
merged 2 commits into from
Dec 14, 2023
Merged

fix: 🐛 broken page #5092

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d9a5ea
fix: 🐛 broken page
jaskaransarkaria Dec 13, 2023
71ee0e0
Commit changes made by code formatters
github-actions[bot] Dec 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions runbooks/source/node-group-changes.html.md.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,34 @@
---
title: Handling Node Group and Instance Changes
weight: 54
last_reviewed_on: 2023-13-12
last_reviewed_on: 2023-12-13
review_in: 6 months
---

# <%= current_page.data.title %>
# Making changes to EKS node groups or instances types

## Why?

You may need to make a change to an EKS [cluster-node-group] or [instance-type-config]. We can't just let terraform apply these changes because terraform doesn't gracefully rollout the old and new nodes. Terraform will bring down all of the old nodes immediately, which will cause outages to users.
You may need to make a change to an EKS [cluster node group] or [instance type config]. We can't just let terraform apply these changes because terraform doesn't gracefully rollout the old and new nodes. Terraform will bring down all of the old nodes immediately, which will cause outages to users.

## How?

The method to avoid bringing down all the nodes at once is to follow these steps:

1. add a new node group with your [updated-changes]
1. add a new node group with your [updated changes]
1. lookup the old node group name (you can find this in the aws gui)
1. once merged in you can drain the old node group using the following command:
> cloud-platform pipeline cordon-and-drain --cluster-name <cluster_name> --node-group <old_node_group_name>
1. raise a new [pr-deleting] the old node group
1. once merged in you can drain the old node group using the command below:
1. raise a new [pr deleting] the old node group

> ```cloud-platform pipeline cordon-and-drain --cluster-name <cluster_name> --node-group <old_node_group_name>```

notes:

- When making changes to the default node group in live, it's handy to pause the pipelines for each of our environments for the duration of the change.
- the `cloud-platform pipeline` command [cordons-and-drains-nodes] in a given node group waiting 2mins between each drained node. This command runs remotely in concourse.

[cluster-node-group]: https://github.com/ministryofjustice/cloud-platform-infrastructure/blob/97768bfd8b4e25df6f415035acac60cf531d88c1/terraform/aws-accounts/cloud-platform-aws/vpc/eks/cluster.tf#L60
[instance-type-config]: https://github.com/ministryofjustice/cloud-platform-infrastructure/blob/97768bfd8b4e25df6f415035acac60cf531d88c1/terraform/aws-accounts/cloud-platform-aws/vpc/eks/cluster.tf#L43
[pr-deleting]: https://github.com/ministryofjustice/cloud-platform-infrastructure/pull/2663
[updated-changes]: https://github.com/ministryofjustice/cloud-platform-infrastructure/pull/2657
[cordons-and-drains-nodes]: https://github.com/ministryofjustice/cloud-platform-terraform-concourse/blob/main/pipelines/manager/main/cordon-and-drain-nodes.yaml
[cluster node group]: https://github.com/ministryofjustice/cloud-platform-infrastructure/blob/97768bfd8b4e25df6f415035acac60cf531d88c1/terraform/aws-accounts/cloud-platform-aws/vpc/eks/cluster.tf#L60
[instance type config]: https://github.com/ministryofjustice/cloud-platform-infrastructure/blob/97768bfd8b4e25df6f415035acac60cf531d88c1/terraform/aws-accounts/cloud-platform-aws/vpc/eks/cluster.tf#L43
[pr deleting]: https://github.com/ministryofjustice/cloud-platform-infrastructure/pull/2663
[updated changes]: https://github.com/ministryofjustice/cloud-platform-infrastructure/pull/2657
[cordons and drains nodes]: https://github.com/ministryofjustice/cloud-platform-terraform-concourse/blob/main/pipelines/manager/main/cordon-and-drain-nodes.yaml