Skip to content

Commit

Permalink
Merge pull request #6129 from ministryofjustice/runbooks-update
Browse files Browse the repository at this point in the history
update review dates, modify punctuations and tenses
  • Loading branch information
FolarinOyenuga authored Sep 3, 2024
2 parents 51a9ed0 + ad283c7 commit c598864
Show file tree
Hide file tree
Showing 5 changed files with 27 additions and 27 deletions.
12 changes: 6 additions & 6 deletions runbooks/source/aws-leaked-credentials.html.md.erb
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@
---
title: AWS Compromised Credentials
weight: 110
last_reviewed_on: 2024-02-23
last_reviewed_on: 2024-09-03
review_in: 6 months
---

# AWS Compromised Credentials

This article is the immediate response, it was created to minimise our window of exposure
This article was created to minimise our window of exposure.

## Steps for a leaked credentials

1) Login into our AWS Management Console

2) Go to **Services -> IAM** and search for the user with credentials leaked (**TIP** You can search either by *Access Keys* or by *Username*). In case you prefer using the CLI:
2) Go to **Services -> IAM** and search for the user with the leaked credentials (**TIP** You can search either by *Access Keys* or by *Username*). In case you prefer using the CLI:

```
aws iam list-users --output json --query 'Users[?contains(UserName, `testAlejandro`) == `true`]'
```

3) Within the User, click in "Delete user" button (top right corner). If you prefer to use the CLI you'll have to delete dependencies (Access Keys, Groups, etc), an example can be found [here](https://gist.github.com/kgmoore431/886aa60cd7fdc9a43bc5c05f9e956adf).
3) Within the User, click on "Delete user" button (top right corner). If you prefer to use the CLI, you'll have to delete dependencies (Access Keys, Groups, etc), an example can be found [here](https://gist.github.com/kgmoore431/886aa60cd7fdc9a43bc5c05f9e956adf).

4) If the service team is known we should notify them via their slack channel (and @ the specific user). If the service team isn't known message should be sent privately on Slack.
4) If the service team is known, we should notify them via their slack channel (and @ the specific user). If the service team isn't known, messages should be sent privately on Slack.

## Getting new credentials

Expand All @@ -30,4 +30,4 @@ Most of the users and keys are created through terraform, the process to recreat
## Audit the compromised credentials

Check CloudTrail for any activity of the credentials after it got exposed. This can be done by logging into AWS Management Console
Go to Services -> CloudTrail -> Event history and filter by `AWS access key`
Go to Services -> CloudTrail -> Event history and filter by `AWS access key`.
26 changes: 13 additions & 13 deletions runbooks/source/cloud-platform-communications-plan.html.md.erb
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: Cloud Platform Communications Plan
weight: 9190
last_reviewed_on: 2024-02-23
last_reviewed_on: 2024-09-03
review_in: 6 months
---

# Cloud Platform Communications Plan

This plan details to the who, how and when we will communicate information to users and stakeholders.
This plan details the who, how and when we will communicate information to users and stakeholders.

## The Plan

Expand All @@ -20,8 +20,8 @@ The table below sets out the typical types of communications the Cloud Platform
| What we are working on now | Service Teams | Fortnightly | #cloud-platform-update |
| Things we have learned | Justice Digital & Technology | Regularly when we have key things to show or promote | Show the Thing |
| Service impacting incidents | Service Teams | When an incident has been declared, key updates during resolution, and at incident closure | #cloud-platform-update |
| Sharing postmortems for service impacting incidents | Service Teams | When postmortem has been documented | #cloud-platform-update |
| Service impacting upgrades/Maintenance | Service Teams | As required | #cloud-platform-update |
| Sharing postmortems for service-impacting incidents | Service Teams | When postmortem has been documented | #cloud-platform-update |
| Service-impacting upgrades/Maintenance | Service Teams | As required | #cloud-platform-update |
| Sharing successes | Chief Technology Officer | Weekly | One to ones and team meetings |
| Sharing successes | SMT | Fortnightly | Architecture & Platforms weeknotes |
| Sharing successes | Justice Digital & Technology | Regularly when we have key things to show or promote | Post about them in #chat or #announcements on Slack/Consider for a Show the Thing |
Expand All @@ -32,7 +32,7 @@ The #cloud-platform-update channel is used for a number of different types of co

### Examples

> **Kubernetes 1.14 Upgrade**
> **Kubernetes 1.27 Upgrade**

> **Incident - Sentry unavailable**

Expand All @@ -49,21 +49,21 @@ The #cloud-platform-update channel is used for a number of different types of co
### Example

> **High Priority Incident Declared - Cloud Platform**
> We are aware that some users are experiencing issues with the access to services on the Cloud Platform this morning.
> We are aware that some users are experiencing issues with access to services on the Cloud Platform this morning.
> We are unsure of the full impact of these issues and the extent to which this is impacting services.
> An incident team has been formed and the team are investigating.
> We will provide further updates in due course, but in any event the next update will be in 30 minutes.
> An incident team has been formed and the team is investigating.
> We will provide further updates in due course, but in any event, the next update will be in 30 minutes.
> Thank you for your patience.

### Things to include in upgrade communications

- What you are upgrading
- When the upgrade will take place (including times)
- Why you are upgrading (in might be useful to include a link to some change notes or something that users can refer to if they want more information rather than adding it all in the comms.)
- Why you are upgrading (it might be useful to include a link to some change notes or something that users can refer to if they want more information rather than adding it all in the comms.)
- Details of any service impact or anything users might need to take action on as a result of changes (including if we are pausing pipelines)
- Details of any risks posed to services as a result of not taking requested actions
- Include examples of code if it helps users understand the changes
- Provide a high level overview of the process we will be taking to implement changes
- Provide a high-level overview of the process we will be taking to implement changes
- Refer users back to the #ask-cloud-platform channel if they have questions or something isn't working as expected

### Example
Expand All @@ -88,10 +88,10 @@ The #cloud-platform-update channel is used for a number of different types of co
> **Downtime**:
> We are not expecting any downtime as a result of this change.
>
> **Process**: (overview of steps that need to be taken my users and the the Cloud Platform Team)
> **Process**: (overview of steps that need to be taken by users and the the Cloud Platform Team)
>
> If you experience any issues during or after the changes implemented please contact the team over at #ask-cloud-platform
> If you experience any issues during or after the changes are implemented, please contact the team over at #ask-cloud-platform

## Sharing information with the wider Ministry of Justice and the Public

There may be occasions where we want to publish something to the wider MoJ or the Public. In these instances we can publish something on the MoJ Digital blog. Requests to publish on the blog should be made to #ask-comms in the first instance.
There may be occasions where we want to publish something to the wider MoJ or the Public. In these instances, we can publish something on the MoJ Digital blog. Requests to publish on the blog should be made to #ask-comms in the first instance.
2 changes: 1 addition & 1 deletion runbooks/source/custom-domain.html.md.erb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Add a custom domain
weight: 80
last_reviewed_on: 2024-02-23
last_reviewed_on: 2024-09-03
review_in: 6 months
---

Expand Down
10 changes: 5 additions & 5 deletions runbooks/source/incident-process.html.md.erb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Incident Process
weight: 40
last_reviewed_on: 2024-02-23
last_reviewed_on: 2024-09-03
review_in: 6 months
---

Expand All @@ -20,9 +20,9 @@ We define an incident as an event which:
* degrades user-facing services, or
* increases risk to production services

> "Users" includes end-users of services (citizens or members of internal user groups such as prison officers), as well as users of the platform - i.e. members of service teams who maintain or depend on services we host/maintain.
> "Users" include end-users of services (citizens or members of internal user groups such as prison officers), as well as users of the platform - i.e. members of service teams who maintain or depend on services we host/maintain.

> An example of increased risk might be when one or more members of a high-availability set of components stops working. e.g. if one of our three master nodes in live-1 stopped working, it would not have any visible effect to end users but the cluster would be at increased risk, because we would no longer have a highly-available cluster.
> An example of increased risk might be when one or more members of a high-availability set of components stop working. e.g. if one of our three master nodes in live-1 stopped working, it would not have any visible effect to end users but the cluster would be at increased risk, because we would no longer have a highly-available cluster.

If this event does not constitute an incident, the appropriate response is probably to [raise a ticket] to fix whatever needs fixing.

Expand Down Expand Up @@ -109,7 +109,7 @@ Once appointed, the scribe should post this message on the incident slack thread
I am the scribe
```

The form of the log is at the scribe's discretion, provided key events are timestamped, and that it can easily be handed off to another member of the team if they take over as scribe.
The form of the log is at the scribe's discretion, provided key events are timestamped, and that it can be easily handed off to another member of the team if they take over as scribe.

### 3.3 Communications Lead

Expand All @@ -129,7 +129,7 @@ People to update:
* Team members for awareness or because they might be able to help - via #cloud-platform
* People in the team who manage communication with senior leadership in MoJ - Steve, Karen, Tony.

In the case of high-priority user-impacting incidents there is a need to keep the MoJ Incident Management Team aware. This is done by posting updates in the private **#p1s** slack channel (only Steve and Tony can do this), and via email to **[[email protected]](mailto:[email protected])**
In the case of high-priority user-impacting incidents, there is a need to keep the MoJ Incident Management Team aware. This is done by posting updates in the private **#p1s** slack channel (only Steve and Tony can do this), and via email to **[[email protected]](mailto:[email protected])**

### Transferring roles

Expand Down
4 changes: 2 additions & 2 deletions runbooks/source/rotate-git-crypt-key.html.md.erb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Git-crypt
weight: 75
last_reviewed_on: 2024-02-23
last_reviewed_on: 2024-09-03
review_in: 6 months
---

Expand Down Expand Up @@ -127,6 +127,6 @@ git show HEAD:<path-to-file>

- where `<path-to-file>` is either absolute from the base of the git repo or relative (eg.: `git show:./my-secret-file.yaml`)

- You should see binary output which begins with `^@GITCRYPT^@`.
- You should see a binary output which begins with `^@GITCRYPT^@`.

Note: If you need to `checkout` an older commit, branch, tag etc., make sure to `git-crypt lock` your repository beforehand in order to avoid a broken local working directory. Once you've locked and checked out the desired revision, you can `git-crypt unlock`.

0 comments on commit c598864

Please sign in to comment.