ci: collect cluster metrics to OpenSearch #2347

msanft · 2023-09-19T14:12:18Z

Context

Previously, no metrics were collected in our E2E tests. This hinders us to track regressions in Constellation as well as debugging problems, such as the recent etcd timeouts.

Proposed change(s)

Deploy Metricbeat and make it available both as part of our debugd-logcollection deployment as well as the K8s deployment that is deployed to the clusters in our E2E tests.
TODO: Discuss which modules make sense in which collection periods

Additional info

Associated Fix PR with E2E runs: ci: fix debugd logcollection #2355
E2E Test

Checklist

Update docs
Add labels (e.g., for changelog category)
Is PR title adequate for changelog?
Link to Milestone

netlify · 2023-09-19T14:12:39Z

✅ Deploy Preview for constellation-docs ready!

Name	Link
🔨 Latest commit	`a967fcc`
🔍 Latest deploy log	https://app.netlify.com/sites/constellation-docs/deploys/65141e5ec36eac00099d9939
😎 Deploy Preview	https://deploy-preview-2347--constellation-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

malt3

I mostly looked at the changed made to debugd

3u13r

LGTM

Signed-off-by: Moritz Sanft <[email protected]>

* add missing keyvault access role Signed-off-by: Moritz Sanft <[email protected]> * bump logstash image version Signed-off-by: Moritz Sanft <[email protected]> * bump filebeat / metricbeat image version Signed-off-by: Moritz Sanft <[email protected]> * log used image version Signed-off-by: Moritz Sanft <[email protected]> * use debugging image versions Signed-off-by: Moritz Sanft <[email protected]> * increase wait timeout for image upload * add cloud metadata processor to filebeat Signed-off-by: Moritz Sanft <[email protected]> * fix template locations in container Signed-off-by: Moritz Sanft <[email protected]> * fix image version typo Signed-off-by: Moritz Sanft <[email protected]> * add filebeat / metricbeat users Signed-off-by: Moritz Sanft <[email protected]> * remove user additions Signed-off-by: Moritz Sanft <[email protected]> * update workflow step name Signed-off-by: Moritz Sanft <[email protected]> * only mount config files Signed-off-by: Moritz Sanft <[email protected]> * document potential rc Signed-off-by: Moritz Sanft <[email protected]> * fix IAM permissions in workflow Signed-off-by: Moritz Sanft <[email protected]> * fix AWS permissions Signed-off-by: Moritz Sanft <[email protected]> * tidy Signed-off-by: Moritz Sanft <[email protected]> * add missing workflow input Signed-off-by: Moritz Sanft <[email protected]> * rename action Signed-off-by: Moritz Sanft <[email protected]> * pin image versions Signed-off-by: Moritz Sanft <[email protected]> * remove unnecessary workflow inputs Signed-off-by: Moritz Sanft <[email protected]> --------- Signed-off-by: Moritz Sanft <[email protected]>

Signed-off-by: Moritz Sanft <[email protected]>

internal/osimage/aws/awsupload.go

debugd/metricbeat/templates/metricbeat.yml

Signed-off-by: Moritz Sanft <[email protected]>

github-actions · 2023-09-27T12:26:56Z

Coverage report

Package	Old	New	Trend
debugd/filebeat	[no test files]	[no test files]	🚧
debugd/internal/debugd/logcollector	13.70%	12.70%	↘️
debugd/metricbeat	0.00%	[no test files]	🚨
hack/logcollector/cmd	[no test files]	[no test files]	🚧
hack/logcollector/internal	[no test files]	[no test files]	🚧
internal/osimage/aws	[no test files]	[no test files]	🚧
internal/versions	8.20%	8.20%	🚧

* add Metricbeat deployment to debugd Signed-off-by: Moritz Sanft <[email protected]> * set metricbeat debugd image version Signed-off-by: Moritz Sanft <[email protected]> * fix k8s deployment Signed-off-by: Moritz Sanft <[email protected]> * use 2 separate deployments Signed-off-by: Moritz Sanft <[email protected]> * only deploy via k8s in non-debug-images Signed-off-by: Moritz Sanft <[email protected]> * add missing tilde * remove k8s metrics Signed-off-by: Moritz Sanft <[email protected]> * unify flag Signed-off-by: Moritz Sanft <[email protected]> * add cloud metadata processor to filebeat Signed-off-by: Moritz Sanft <[email protected]> * ci: fix debugd logcollection (#2355) * add missing keyvault access role Signed-off-by: Moritz Sanft <[email protected]> * bump logstash image version Signed-off-by: Moritz Sanft <[email protected]> * bump filebeat / metricbeat image version Signed-off-by: Moritz Sanft <[email protected]> * log used image version Signed-off-by: Moritz Sanft <[email protected]> * use debugging image versions Signed-off-by: Moritz Sanft <[email protected]> * increase wait timeout for image upload * add cloud metadata processor to filebeat Signed-off-by: Moritz Sanft <[email protected]> * fix template locations in container Signed-off-by: Moritz Sanft <[email protected]> * fix image version typo Signed-off-by: Moritz Sanft <[email protected]> * add filebeat / metricbeat users Signed-off-by: Moritz Sanft <[email protected]> * remove user additions Signed-off-by: Moritz Sanft <[email protected]> * update workflow step name Signed-off-by: Moritz Sanft <[email protected]> * only mount config files Signed-off-by: Moritz Sanft <[email protected]> * document potential rc Signed-off-by: Moritz Sanft <[email protected]> * fix IAM permissions in workflow Signed-off-by: Moritz Sanft <[email protected]> * fix AWS permissions Signed-off-by: Moritz Sanft <[email protected]> * tidy Signed-off-by: Moritz Sanft <[email protected]> * add missing workflow input Signed-off-by: Moritz Sanft <[email protected]> * rename action Signed-off-by: Moritz Sanft <[email protected]> * pin image versions Signed-off-by: Moritz Sanft <[email protected]> * remove unnecessary workflow inputs Signed-off-by: Moritz Sanft <[email protected]> --------- Signed-off-by: Moritz Sanft <[email protected]> * add refStream input Signed-off-by: Moritz Sanft <[email protected]> * remove inputs.yml dep Signed-off-by: Moritz Sanft <[email protected]> * increase system metric period Signed-off-by: Moritz Sanft <[email protected]> * fix linkchecker Signed-off-by: Moritz Sanft <[email protected]> --------- Signed-off-by: Moritz Sanft <[email protected]>

msanft added the no changelog Change won't be listed in release changelog label Sep 19, 2023

msanft added this to the v2.12.0 milestone Sep 19, 2023

msanft requested a review from katexochen September 19, 2023 14:12

katexochen removed their request for review September 19, 2023 14:13

msanft requested a review from derpsteb September 20, 2023 09:17

msanft marked this pull request as ready for review September 20, 2023 09:17

msanft requested review from 3u13r and malt3 as code owners September 20, 2023 09:17

msanft marked this pull request as draft September 20, 2023 11:58

msanft marked this pull request as ready for review September 25, 2023 08:54

msanft requested a review from daniel-weisse September 26, 2023 07:22

malt3 approved these changes Sep 26, 2023

View reviewed changes

daniel-weisse approved these changes Sep 26, 2023

View reviewed changes

3u13r approved these changes Sep 26, 2023

View reviewed changes

msanft added 10 commits September 26, 2023 15:14

add Metricbeat deployment to debugd

2c22f1f

Signed-off-by: Moritz Sanft <[email protected]>

set metricbeat debugd image version

7210389

Signed-off-by: Moritz Sanft <[email protected]>

fix k8s deployment

954b0ed

Signed-off-by: Moritz Sanft <[email protected]>

use 2 separate deployments

5b67554

Signed-off-by: Moritz Sanft <[email protected]>

only deploy via k8s in non-debug-images

bb8426c

Signed-off-by: Moritz Sanft <[email protected]>

add missing tilde

34594b8

remove k8s metrics

3c69d09

Signed-off-by: Moritz Sanft <[email protected]>

unify flag

42fef6a

Signed-off-by: Moritz Sanft <[email protected]>

add cloud metadata processor to filebeat

40fbe5b

Signed-off-by: Moritz Sanft <[email protected]>

msanft force-pushed the feat/debugd/metricbeat-deployment branch from 4b5cf09 to 67ec24b Compare September 26, 2023 13:19

add refStream input

91494bc

Signed-off-by: Moritz Sanft <[email protected]>

msanft had a problem deploying to e2e September 26, 2023 13:22 — with GitHub Actions Failure

remove inputs.yml dep

53994b0

Signed-off-by: Moritz Sanft <[email protected]>

derpsteb reviewed Sep 27, 2023

View reviewed changes

internal/osimage/aws/awsupload.go Show resolved Hide resolved

derpsteb reviewed Sep 27, 2023

View reviewed changes

debugd/metricbeat/templates/metricbeat.yml Show resolved Hide resolved

derpsteb approved these changes Sep 27, 2023

View reviewed changes

msanft added 2 commits September 27, 2023 14:15

increase system metric period

e743df8

Signed-off-by: Moritz Sanft <[email protected]>

fix linkchecker

a967fcc

Signed-off-by: Moritz Sanft <[email protected]>

msanft requested a review from thomasten as a code owner September 27, 2023 12:21

msanft merged commit f4b2d02 into main Sep 27, 2023
12 checks passed

msanft deleted the feat/debugd/metricbeat-deployment branch September 27, 2023 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: collect cluster metrics to OpenSearch #2347

ci: collect cluster metrics to OpenSearch #2347

msanft commented Sep 19, 2023 •

edited

Loading

netlify bot commented Sep 19, 2023 •

edited

Loading

malt3 left a comment

3u13r left a comment

github-actions bot commented Sep 27, 2023

ci: collect cluster metrics to OpenSearch #2347

ci: collect cluster metrics to OpenSearch #2347

Conversation

msanft commented Sep 19, 2023 • edited Loading

Context

Proposed change(s)

Additional info

Checklist

netlify bot commented Sep 19, 2023 • edited Loading

✅ Deploy Preview for constellation-docs ready!

malt3 left a comment

Choose a reason for hiding this comment

3u13r left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 27, 2023

Coverage report

msanft commented Sep 19, 2023 •

edited

Loading

netlify bot commented Sep 19, 2023 •

edited

Loading