MCO: add service monitor to alertmanager #1319

thibaultmg · 2024-02-09T10:17:47Z

Adds platform serviceMonitor for alertManager following platform monitoring recommendations:

Adds serviceMonitor configured with tls
Adds kube-rbac-proxy sidecar for protecting the metrics endpoint, exposes it on port 9096
Removes exposed 9093 port from alertmanager service
Adds new service for port 9096 because existing services have incompatible settings for the metrics endpoint needs (sessionAffinity)
Adds rbac for fetching configmap in kube-system namespace
Adds watcher for the configmap in kube-system

As a side note, kube-rbac-proxy automatically reloads TLS and client CA files. So no need to restart the service when they change.

I have tested the manifests and the new serviceMonitor is well being picked up by platform monitoring, and it is able to fetch those new metrics.

Jira task: https://issues.redhat.com/browse/RHOBS-1002

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

openshift-ci · 2024-02-09T10:17:53Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

philipgough · 2024-02-13T11:22:26Z

/test test-unit

douglascamata · 2024-02-13T11:22:54Z

operators/multiclusterobservability/manifests/base/alertmanager/alertmanager-operated.yaml

-  namespace: open-cluster-management
+  namespace: open-cluster-management-observability 


Is this change necessary? Is it breaking? 🤔

All those namespaces are reset by the controller. So personally, I would remove all of them from the manifests. But I am not sure it has not impact 🤷
Resetting this one as it wasn't changed on purpose

I removed all unnecessary namespaces. Unit tests are checking them now

operators/multiclusterobservability/manifests/base/alertmanager/alertmanager-statefulset.yaml

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

douglascamata · 2024-02-13T18:01:53Z

@thibaultmg something's badly wrong with Prow, it seems.

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

…icemonitor Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

operators/multiclusterobservability/pkg/rendering/renderer_alertmanager.go

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

…icemonitor

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

…icemonitor Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

douglascamata

LGTM

douglascamata · 2024-02-28T15:15:18Z

...ticlusterobservability/manifests/base/alertmanager/alertmanager-accessor-serviceaccount.yaml

@@ -2,6 +2,5 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
  name: observability-alertmanager-accessor
-  namespace: open-cluster-management


Is this expected? In which namespace will this land on?

Service account resources are handled by "ServiceAccount": r.renderer.RenderNamespace,
So it will land as before in the namespace := mcoconfig.GetDefaultNamespace() which is open-cluster-management-observability
I removed all of them so that there is no ambiguity. And they are all checked in unit tests.

douglascamata · 2024-02-28T15:22:54Z

/hold

openshift-ci · 2024-02-28T19:42:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: douglascamata, thibaultmg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~operators/multiclusterobservability/OWNERS~~ [douglascamata,thibaultmg]
~~tests/OWNERS~~ [douglascamata,thibaultmg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

douglascamata · 2024-02-29T10:06:44Z

/hold cancel

…icemonitor

openshift-ci · 2024-03-01T09:06:50Z

New changes are detected. LGTM label has been removed.

openshift-ci · 2024-03-01T09:06:55Z

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.

📝 Please follow instructions in the contributing guide to update your commits with the DCO

Full details of the Developer Certificate of Origin can be found at developercertificate.org.

The list of commits missing DCO signoff:

8c8398f Merge branch 'main' into add_alertmanager_servicemonitor
de98b30 Merge remote-tracking branch 'origin/main' into add_alertmanager_servicemonitor
cb87dea Merge remote-tracking branch 'origin/main' into add_alertmanager_servicemonitor
c965b32 Merge remote-tracking branch 'origin/main' into add_alertmanager_servicemonitor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sonarqubecloud · 2024-03-01T09:14:10Z

Quality Gate failed

Failed conditions
56.7% Coverage on New Code (required ≥ 70%)
C Security Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

openshift-ci · 2024-03-01T09:14:17Z

@thibaultmg: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-kind	`4b3d030`	link	true	`/test e2e-kind`
ci/prow/test-e2e	`4b3d030`	link	true	`/test test-e2e`
ci/prow/sonarcloud	`c965b32`	link	true	`/test sonarcloud`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

* init Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add cm watcher Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * cleaning Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * change rbac-proxy image Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix namespaces names Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix unit tests Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix serviceMonitor namespace Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add e2e test debug info Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add mch-image-manifest debug info Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve image replacement logging Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix call to image replacer Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add service monitor crd to kind e2e Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add debug logging oba e2e tests Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve logs, reduce volume Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve ci logs Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve ci logs Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add oba logs Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve oba logging Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add unit tests Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add apiextensionsv1 scheme Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add resource version to crd update Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> Signed-off-by: Coleen Iona Quadros <coleen.quadros27@gmail.com>

* init Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add cm watcher Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * cleaning Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * change rbac-proxy image Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix namespaces names Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix unit tests Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix serviceMonitor namespace Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add e2e test debug info Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add mch-image-manifest debug info Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve image replacement logging Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix call to image replacer Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add service monitor crd to kind e2e Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add debug logging oba e2e tests Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve logs, reduce volume Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve ci logs Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve ci logs Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * fix Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add oba logs Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * improve oba logging Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add unit tests Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add apiextensionsv1 scheme Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> * add resource version to crd update Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com> --------- Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

init

86ee16e

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

openshift-ci bot added the do-not-merge/work-in-progress label Feb 9, 2024

openshift-ci bot added dco-signoff: yes approved labels Feb 9, 2024

coleenquadros marked this pull request as ready for review February 12, 2024 09:15

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 12, 2024

coleenquadros added the do-not-merge/hold label Feb 12, 2024

coleenquadros marked this pull request as draft February 12, 2024 10:01

openshift-ci bot added the do-not-merge/work-in-progress label Feb 12, 2024

thibaultmg added 2 commits February 13, 2024 11:53

add cm watcher

3f391e9

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

cleaning

22af01a

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

thibaultmg marked this pull request as ready for review February 13, 2024 11:12

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 13, 2024

thibaultmg enabled auto-merge (squash) February 13, 2024 11:13

douglascamata suggested changes Feb 13, 2024

View reviewed changes

openshift-ci bot assigned douglascamata Feb 13, 2024

thibaultmg added 2 commits February 13, 2024 17:13

change rbac-proxy image

a9ac90d

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

fix namespaces names

b6dd002

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

thibaultmg requested a review from douglascamata February 13, 2024 16:31

Merge branch 'main' into add_alertmanager_servicemonitor

8c8398f

openshift-ci bot added dco-signoff: no and removed dco-signoff: yes labels Feb 14, 2024

thibaultmg added 5 commits February 14, 2024 10:57

fix unit tests

7663e72

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

fix serviceMonitor namespace

8d6637f

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

add e2e test debug info

5cab312

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

add mch-image-manifest debug info

73fe43c

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

fix

e7a5286

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

openshift-merge-robot removed the needs-rebase label Feb 16, 2024

thibaultmg added 3 commits February 16, 2024 17:40

fix

72cf79e

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into add_alertmanager_serv…

18be7a6

…icemonitor Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

add oba logs

97ba5c0

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

saswatamcode reviewed Feb 21, 2024

View reviewed changes

operators/multiclusterobservability/pkg/rendering/renderer_alertmanager.go Show resolved Hide resolved

thibaultmg added 7 commits February 22, 2024 09:08

improve oba logging

d47ec06

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

add unit tests

3b48110

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into add_alertmanager_serv…

de98b30

…icemonitor

Merge remote-tracking branch 'origin/main' into add_alertmanager_serv…

cb87dea

…icemonitor

add apiextensionsv1 scheme

066c267

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

add resource version to crd update

f8b479a

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into add_alertmanager_serv…

4b3d030

…icemonitor Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

douglascamata approved these changes Feb 28, 2024

View reviewed changes

openshift-ci bot added the lgtm label Feb 28, 2024

douglascamata approved these changes Feb 28, 2024

View reviewed changes

openshift-ci bot removed the do-not-merge/hold label Feb 29, 2024

Merge remote-tracking branch 'origin/main' into add_alertmanager_serv…

c965b32

…icemonitor

openshift-ci bot removed the lgtm label Mar 1, 2024

thibaultmg disabled auto-merge March 1, 2024 09:27

thibaultmg merged commit b265920 into stolostron:main Mar 1, 2024
8 of 14 checks passed

thibaultmg deleted the add_alertmanager_servicemonitor branch March 1, 2024 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MCO: add service monitor to alertmanager #1319

MCO: add service monitor to alertmanager #1319

thibaultmg commented Feb 9, 2024 •

edited

Loading

openshift-ci bot commented Feb 9, 2024

philipgough commented Feb 13, 2024

douglascamata Feb 13, 2024

thibaultmg Feb 13, 2024

thibaultmg Feb 22, 2024

douglascamata commented Feb 13, 2024

douglascamata left a comment

douglascamata Feb 28, 2024 •

edited

Loading

thibaultmg Feb 28, 2024

douglascamata commented Feb 28, 2024

openshift-ci bot commented Feb 28, 2024

douglascamata commented Feb 29, 2024

openshift-ci bot commented Mar 1, 2024

openshift-ci bot commented Mar 1, 2024

sonarqubecloud bot commented Mar 1, 2024

openshift-ci bot commented Mar 1, 2024

		namespace: open-cluster-management
		namespace: open-cluster-management-observability

MCO: add service monitor to alertmanager #1319

MCO: add service monitor to alertmanager #1319

Conversation

thibaultmg commented Feb 9, 2024 • edited Loading

openshift-ci bot commented Feb 9, 2024

philipgough commented Feb 13, 2024

douglascamata Feb 13, 2024

Choose a reason for hiding this comment

thibaultmg Feb 13, 2024

Choose a reason for hiding this comment

thibaultmg Feb 22, 2024

Choose a reason for hiding this comment

douglascamata commented Feb 13, 2024

douglascamata left a comment

Choose a reason for hiding this comment

douglascamata Feb 28, 2024 • edited Loading

Choose a reason for hiding this comment

thibaultmg Feb 28, 2024

Choose a reason for hiding this comment

douglascamata commented Feb 28, 2024

openshift-ci bot commented Feb 28, 2024

douglascamata commented Feb 29, 2024

openshift-ci bot commented Mar 1, 2024

openshift-ci bot commented Mar 1, 2024

sonarqubecloud bot commented Mar 1, 2024

Quality Gate failed

openshift-ci bot commented Mar 1, 2024

thibaultmg commented Feb 9, 2024 •

edited

Loading

douglascamata Feb 28, 2024 •

edited

Loading