Check Pod Autodiscover conditions for node/namespace updates

[gizas]

Context

This issue tries to resolve the #34717
For more details, upon of Node or Namespace event updates, we trigger updates for all pods that are related to the specific resources.

The issue is that many of those updates are not related with the actual metadata information that the watchers are made for. So we need to check which of those updates are the ones that should trigger pod watcher restarts

Solution

We enhance the nodePodupdater function and we also pass the nodeWatcher as a parameter https://github.com/elastic/beats/blob/conditionsonPodupdater/libbeat/autodiscover/providers/kubernetes/pod.go#L156

Then we are able to check whether those updates are releavnt or not in order to trigger a pod update in https://github.com/elastic/elastic-agent-autodiscover/blob/conditionsonPodupdater/kubernetes/eventhandler.go#L236-L245

How to Test

• Create a kind cluster kind create cluster
• Create a local stack with elastic-package stack up -d -vvv --version=8.12.0-SNAPSHOT
• Follow Process (just replace metricbeat with filebeat)
• Install https://github.com/elastic/beats/blob/main/filebeat/_meta/test/docs/01_playground/filebeat.yaml
• Download the elastic-agent-autodiscover repo linked in this story with relevant updates. In order to make filebeat to use the local copy of your elastic-agent-autodiscover library, update the /beats/go.mod file:

replace github.com/elastic/elastic-agent-autodiscover => /Users/andreasgkizas/elastic/elastic-agent-autodiscover

• Build filebeat GOOS=linux GOARCH=arm64 go build
• You can monitor node updates on the background

kubectl proxy &
stdbuf -oL curl -s '127.0.0.1:8001/api/v1/nodes?allowWatchBookmarks=true&watch=true&pretty=false' > nodes.log &

• Trigger a node with eg.kubectl label nodes kind-control-plane katsoulis=pasole

Definition of Done

• Test for a longer period of time for node updates
• Repeat same procedure on namespacePodUpdater
• Test and verify that metricbeat is not affected. (Reason is to see how autodiscover affects metricbeat)
• Test with a full Elastic Agent build to see if something is not broken

Testing

1. Updating label of namespace

kubectl label namespaces default katsoulis3=pasole

2. Updating label of node

kubectl label node kind-control-plane pas1=pas1

3. Checking the events of nodes

Metadata Event update in node triggers pod update

Healthcheck Event update in node does not trigger pod update

Relevant link:

• Repo in elastic-agent-autodiscover
• Beats Repo

Tasks

Preview Give feedback

1. PR in elastic-agent-autodiscover/
2. PR in beats

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[constanca-m]

I ran an environment with these changes.

When I did:

kubectl label node kind-control-plane teste2=teste1

I could see the output:

I left filebeat running for longer, and I could not see more updates on pods from the node healthcheck. But if I trigger it again by updating the labels of the node, I can see it the message again for pod update.

It seems to work fine!