Upgrade kube-prometheus-stack to 55.8.0

[poornima-krishnasamy]

Background

The kube-prometheus-stack installs several components on monitoring namespace. The helm chart is in version kube-prometheus-stack-41.9.1 and the latest is 55.8.0

Approach

The chart has several major version updates. Check for the changelog if one big jump would cause any issues. Would the stack needs to be in certain version before coming to 55.8.0

The crds are installed seperately from the chart. Check for upgrading the instructions for crds. Should this be done one step at a time or a big jump is ok

Check is there any compatibility conflict with kubernetes version

IMPORTANT:

We also need to switch back over to setting the kube-prometheus-stack chart manage our kube-state-metrics versioning. This was temporarily overridden for 1.26 upgrade work to retain compatibility, see here for the change that was made in the prom values template:
ministryofjustice/cloud-platform-terraform-monitoring@9caded0

Which part of the user docs does this impact

https://github.com/ministryofjustice/cloud-platform-terraform-monitoring

#5149

Communicate changes

• post for #cloud-platform-update
• Weeknotes item
• Show the Thing/P&A All Hands/User CoP
• Announcements channel

Questions / Assumptions

Definition of done

• readme has been updated
• Container images issue have been updated
• another team member has reviewed
• smoke tests are green
• prepare demo for the team

Reference

How to write good user stories

[kyphutruong]

Working branch to upgrade kube-prometheus-stack to 51.1.1

[kyphutruong]

Working branch to upgrade kube-prometheus-stack to 51.10.0

[kyphutruong]

kube-prometheus-stack on manager live and live-2 has been upgraded to 51.10.0

PRs: monitoring and infrastructure

[kyphutruong]

kube-prometheus-stack on manager live and live-2 has been upgraded to 52.1.0

PRs: monitoring and infrastructure

[kyphutruong]

kube-prometheus-stack on manager live and live-2 has been upgraded to 56.21.4

PRs: monitoring and infrastructure

[kyphutruong]

As per issue, we need to switch back over to setting the kube-prometheus-stack chart to manage our kube-state-metrics versioning.

This is now a post EKS 1.27 upgrade task. Issue to action this raised separately here