ci: fix debugd logcollection #2355

msanft · 2023-09-22T15:14:38Z

Context

Logcollection in the debugd was not working for quite some time for various reasons, such as missing IAM permissions and broken container images. No debugd-logs are available for quite some time in OpenSearch.

Proposed change(s)

Patch the permissions of the UAMI in our e2e tests, allowing access to the keyvault containing the OpenSearch credentials.
Fix various bugs in the logcollector images that are deployed by the debugd.

Additional info

Checklist

Update docs
Add labels (e.g., for changelog category)
Is PR title adequate for changelog?
Link to Milestone

Signed-off-by: Moritz Sanft <[email protected]>

github-actions · 2023-09-22T15:20:00Z

Coverage report

Package	Old	New	Trend
debugd/filebeat	[no test files]	[no test files]	🚧
debugd/internal/debugd/logcollector	13.70%	12.70%	↘️
debugd/metricbeat	0.00%	[no test files]	🚨
hack/logcollector/cmd	[no test files]	[no test files]	🚧
hack/logcollector/internal	[no test files]	[no test files]	🚧
internal/osimage/aws	[no test files]	[no test files]	🚧
internal/versions	8.20%	8.20%	🚧

Signed-off-by: Moritz Sanft <[email protected]>

.github/actions/deploy_debugd/action.yml

internal/versions/versions.go

Signed-off-by: Moritz Sanft <[email protected]>

* add missing keyvault access role Signed-off-by: Moritz Sanft <[email protected]> * bump logstash image version Signed-off-by: Moritz Sanft <[email protected]> * bump filebeat / metricbeat image version Signed-off-by: Moritz Sanft <[email protected]> * log used image version Signed-off-by: Moritz Sanft <[email protected]> * use debugging image versions Signed-off-by: Moritz Sanft <[email protected]> * increase wait timeout for image upload * add cloud metadata processor to filebeat Signed-off-by: Moritz Sanft <[email protected]> * fix template locations in container Signed-off-by: Moritz Sanft <[email protected]> * fix image version typo Signed-off-by: Moritz Sanft <[email protected]> * add filebeat / metricbeat users Signed-off-by: Moritz Sanft <[email protected]> * remove user additions Signed-off-by: Moritz Sanft <[email protected]> * update workflow step name Signed-off-by: Moritz Sanft <[email protected]> * only mount config files Signed-off-by: Moritz Sanft <[email protected]> * document potential rc Signed-off-by: Moritz Sanft <[email protected]> * fix IAM permissions in workflow Signed-off-by: Moritz Sanft <[email protected]> * fix AWS permissions Signed-off-by: Moritz Sanft <[email protected]> * tidy Signed-off-by: Moritz Sanft <[email protected]> * add missing workflow input Signed-off-by: Moritz Sanft <[email protected]> * rename action Signed-off-by: Moritz Sanft <[email protected]> * pin image versions Signed-off-by: Moritz Sanft <[email protected]> * remove unnecessary workflow inputs Signed-off-by: Moritz Sanft <[email protected]> --------- Signed-off-by: Moritz Sanft <[email protected]>

* add Metricbeat deployment to debugd Signed-off-by: Moritz Sanft <[email protected]> * set metricbeat debugd image version Signed-off-by: Moritz Sanft <[email protected]> * fix k8s deployment Signed-off-by: Moritz Sanft <[email protected]> * use 2 separate deployments Signed-off-by: Moritz Sanft <[email protected]> * only deploy via k8s in non-debug-images Signed-off-by: Moritz Sanft <[email protected]> * add missing tilde * remove k8s metrics Signed-off-by: Moritz Sanft <[email protected]> * unify flag Signed-off-by: Moritz Sanft <[email protected]> * add cloud metadata processor to filebeat Signed-off-by: Moritz Sanft <[email protected]> * ci: fix debugd logcollection (#2355) * add missing keyvault access role Signed-off-by: Moritz Sanft <[email protected]> * bump logstash image version Signed-off-by: Moritz Sanft <[email protected]> * bump filebeat / metricbeat image version Signed-off-by: Moritz Sanft <[email protected]> * log used image version Signed-off-by: Moritz Sanft <[email protected]> * use debugging image versions Signed-off-by: Moritz Sanft <[email protected]> * increase wait timeout for image upload * add cloud metadata processor to filebeat Signed-off-by: Moritz Sanft <[email protected]> * fix template locations in container Signed-off-by: Moritz Sanft <[email protected]> * fix image version typo Signed-off-by: Moritz Sanft <[email protected]> * add filebeat / metricbeat users Signed-off-by: Moritz Sanft <[email protected]> * remove user additions Signed-off-by: Moritz Sanft <[email protected]> * update workflow step name Signed-off-by: Moritz Sanft <[email protected]> * only mount config files Signed-off-by: Moritz Sanft <[email protected]> * document potential rc Signed-off-by: Moritz Sanft <[email protected]> * fix IAM permissions in workflow Signed-off-by: Moritz Sanft <[email protected]> * fix AWS permissions Signed-off-by: Moritz Sanft <[email protected]> * tidy Signed-off-by: Moritz Sanft <[email protected]> * add missing workflow input Signed-off-by: Moritz Sanft <[email protected]> * rename action Signed-off-by: Moritz Sanft <[email protected]> * pin image versions Signed-off-by: Moritz Sanft <[email protected]> * remove unnecessary workflow inputs Signed-off-by: Moritz Sanft <[email protected]> --------- Signed-off-by: Moritz Sanft <[email protected]> * add refStream input Signed-off-by: Moritz Sanft <[email protected]> * remove inputs.yml dep Signed-off-by: Moritz Sanft <[email protected]> * increase system metric period Signed-off-by: Moritz Sanft <[email protected]> * fix linkchecker Signed-off-by: Moritz Sanft <[email protected]> --------- Signed-off-by: Moritz Sanft <[email protected]>

msanft added 14 commits September 21, 2023 09:03

add missing keyvault access role

6141a8d

Signed-off-by: Moritz Sanft <[email protected]>

bump logstash image version

642c043

Signed-off-by: Moritz Sanft <[email protected]>

bump filebeat / metricbeat image version

9c07d5c

Signed-off-by: Moritz Sanft <[email protected]>

log used image version

750871c

Signed-off-by: Moritz Sanft <[email protected]>

use debugging image versions

d02a357

Signed-off-by: Moritz Sanft <[email protected]>

increase wait timeout for image upload

583115c

add cloud metadata processor to filebeat

7c765b0

Signed-off-by: Moritz Sanft <[email protected]>

fix template locations in container

12b37fd

Signed-off-by: Moritz Sanft <[email protected]>

fix image version typo

c928376

Signed-off-by: Moritz Sanft <[email protected]>

add filebeat / metricbeat users

f709b86

Signed-off-by: Moritz Sanft <[email protected]>

remove user additions

3fac45e

Signed-off-by: Moritz Sanft <[email protected]>

update workflow step name

697922f

Signed-off-by: Moritz Sanft <[email protected]>

only mount config files

dbc6d7c

Signed-off-by: Moritz Sanft <[email protected]>

document potential rc

8bb7aa1

Signed-off-by: Moritz Sanft <[email protected]>

fix IAM permissions in workflow

ece2155

Signed-off-by: Moritz Sanft <[email protected]>

msanft added bug fix Fixing a bug no changelog Change won't be listed in release changelog labels Sep 25, 2023

msanft added this to the v2.12.0 milestone Sep 25, 2023

fix AWS permissions

43453f3

Signed-off-by: Moritz Sanft <[email protected]>

msanft mentioned this pull request Sep 25, 2023

ci: collect cluster metrics to OpenSearch #2347

Merged

4 tasks

tidy

0addaf0

Signed-off-by: Moritz Sanft <[email protected]>

msanft had a problem deploying to e2e September 25, 2023 12:18 — with GitHub Actions Error

add missing workflow input

47d69f2

Signed-off-by: Moritz Sanft <[email protected]>

msanft marked this pull request as ready for review September 25, 2023 14:01

msanft requested review from 3u13r and malt3 as code owners September 25, 2023 14:01

msanft requested a review from daniel-weisse September 25, 2023 14:01

daniel-weisse reviewed Sep 25, 2023

View reviewed changes

.github/actions/deploy_debugd/action.yml Outdated Show resolved Hide resolved

internal/versions/versions.go Outdated Show resolved Hide resolved

rename action

9367fc3

Signed-off-by: Moritz Sanft <[email protected]>

msanft added 2 commits September 26, 2023 08:21

pin image versions

001d441

Signed-off-by: Moritz Sanft <[email protected]>

remove unnecessary workflow inputs

e9e9fce

Signed-off-by: Moritz Sanft <[email protected]>

msanft requested a review from daniel-weisse September 26, 2023 07:04

daniel-weisse approved these changes Sep 26, 2023

View reviewed changes

msanft merged commit 4b5cf09 into feat/debugd/metricbeat-deployment Sep 26, 2023
9 of 11 checks passed

msanft deleted the fix/ci/logcollection-permissions branch September 26, 2023 07:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: fix debugd logcollection #2355

ci: fix debugd logcollection #2355

msanft commented Sep 22, 2023 •

edited

Loading

github-actions bot commented Sep 22, 2023

ci: fix debugd logcollection #2355

ci: fix debugd logcollection #2355

Conversation

msanft commented Sep 22, 2023 • edited Loading

Context

Proposed change(s)

Additional info

Checklist

github-actions bot commented Sep 22, 2023

Coverage report

msanft commented Sep 22, 2023 •

edited

Loading