Merge pull request #4776 from ministryofjustice/update-logging

docs: ✏️ update modsec logging docs

runbooks/source/get-audit-log-from-modsec.html.md.erb

@@ -1,15 +1,15 @@
 ---
 title: Get an audit log from modsec
 weight: 8600
-last_reviewed_on: 2023-05-19
-review_in: 3 months
+last_reviewed_on: 2023-09-22
+review_in: 6 months
 ---
 
-# Opensearch modsec setup
+# OpenSearch modsec setup
 
-We have introduced an opensearch dashboard which collects all modsec logs and has document level security enabled. This means users can only access the logs for the github team they are in (see here for more details)[https://user-guide.cloud-platform.service.justice.gov.uk/documentation/networking/modsecurity.html]. With this feature in place users can self serve and access their own modsec logs. In the case of a rare error, we may shut off logs to the dashboard (from fluent-bit) and then you must use the instructions below to access modsec logs on behalf of the user.
+We have introduced an openSearch dashboard which collects all modsec logs and has document level security enabled. This means users can only access the logs for the github team they are in [see here for more details](https://user-guide.cloud-platform.service.justice.gov.uk/documentation/networking/modsecurity.html). With this feature in place users can self serve and access their own modsec logs. In the case of a rare error and logs aren't flowing to OpenSearch, then you must use the instructions below to access modsec logs on behalf of the user.
 
-## Get an audit log from modsec (when fluent-bit is not pushing to opensearch)
+## Get an audit log from modsec (when fluent-bit is not pushing to OpenSearch)
 
 On occasion users may need you to provide them with audit log information on an modsec event from our ingress-controllers. This information may be sensitive so it can't be placed in our org-wide Elasticsearch cluster. You'll need to fetch this information from the pod that generated the log.
 

runbooks/source/resolve-opensearch-no-logs.html.md.erb

@@ -0,0 +1,24 @@
+---
+title: Resolving no logs in modsec OpenSearch
+weight: 190
+last_reviewed_on: 2023-09-22
+review_in: 6 months
+---
+
+# Modsec logging architecture
+
+[Please read up on the architecture first](https://github.com/ministryofjustice/cloud-platform-terraform-ingress-controller/#cloud-platform-terraform-ingress-controller)
+
+## Deebugging
+
+The first place to look is in the fluent bit side car logs
+
+```
+stern modsec --container flb-modsec-audit-logs -n ingress-controllers
+```
+
+This should give you a clue to the issue. Once you resolve this issue. Logs may not immediately come back, this might be because logs are stuck retrying, if this is the case then restart the deployment
+
+```bash
+kubectl rollout restart deployment/nginx-ingress-modsec-controller -n ingress-controllers
+```

runbooks/source/resolve-opensearch-shard-issues.html.md.erb

@@ -1,22 +1,22 @@
 ---
-title: Resolving Open Search shard problems
+title: Resolving OpenSearch shard problems
 weight: 190
-last_reviewed_on: 2023-08-17
+last_reviewed_on: 2023-09-22
 review_in: 6 months
 ---
 
 # Open search best practices
 
-It's important to follow aws best practices around shard sizing for open search or elastic search, these links are helpful:
+It's important to follow aws best practices around shard sizing for OpenSearch or elastic search, these links are helpful:
 
-* https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html
-* https://repost.aws/knowledge-center/opensearch-rebalance-uneven-shards
-* https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html#bp-sharding
-* https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation/
+* [https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html)
+* [https://repost.aws/knowledge-center/opensearch-rebalance-uneven-shards](https://repost.aws/knowledge-center/opensearch-rebalance-uneven-shards])
+* [https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html#bp-sharding](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html#bp-sharding)
+* [https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation](https://aws.amazon.com/blogs/opensource/open-distro-elasticsearch-shard-allocation)
 
 ## Some general info about shards
 
-Sharding distributes workload cross the cluster nodes. When you push data into the search engine, that data is partitioned into shards and split into primary and replica shards. By default Open Search has 5 primary (independent partitions of a full dataset) and 1 replica shard (a full set copy of the primaries so takes up the same amount of disk space). Replicas provide redundancy and read capacity, it is recommended you have at least 1. Each shard resides on different nodes.
+Sharding distributes workload cross the cluster nodes. When you push data into the search engine, that data is partitioned into shards and split into primary and replica shards. By default OpenSearch has 5 primary (independent partitions of a full dataset) and 1 replica shard (a full set copy of the primaries so takes up the same amount of disk space). Replicas provide redundancy and read capacity, it is recommended you have at least 1. Each shard resides on different nodes.
 
 Optimum shard distribution means shards spread evenly across nodes, if you can achieve this then resource usage across the nodes will also be equal.
 
@@ -50,11 +50,11 @@ run the following to create a shell that you can run commands from:
 kubectl run curl-pod -n <your-namespace> --image="alpine/curl" --restart=Never --rm -it -- /bin/sh
 ```
 
-## Connecting to the Open search api
+## Connecting to the OpenSearch api
 
-Because we have fine-grained access enabled on open search connection isn't based on ip. It's based on SAML. To link your cli with opensearch there is a manual step of adding your aws user arn to the `all_access` opensearh role.
+Because we have fine-grained access enabled on OpenSearch connection isn't based on ip. It's based on SAML. To link your cli with OpenSearch there is a manual step of adding your aws user arn to the `all_access` OpenSearh role.
 
-1. login to the opensearch dashboard using github via saml
+1. login to the OpenSearch dashboard using github via saml
 1. as a webops team member you have permissions to edit roles so head to Security -> Roles -> `all_access` (see screenshot below)
 
 ![opensearch add user screenshot from dashboard](./images/add-user-to-opensearch.png)