Grafana agent is not happy in proxy environment

[QuentinBisson]

Description

Grafana agent is not happy behind a proxy and it's not able to send it's events on leopard

ts=2024-11-12T13:30:31.246223133Z level=info msg="failed to send usage report" retries=0 err="Post "https://stats.grafana.org/agent-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
ts=2024-11-12T13:30:38.212122343Z level=info msg="failed to send usage report" retries=1 err="Post "https://stats.grafana.org/agent-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
ts=2024-11-12T13:30:46.0336601Z level=info msg="failed to send usage report" retries=2 err="Post "https://stats.grafana.org/agent-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
ts=2024-11-12T13:30:48.802753301Z level=warn msg="error sending batch, will retry" component=loki.write.default component=client host= status=-1 tenant=leopard error="Post "/loki/api/v1/push": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

TODO

• Well fix it
• Check if it's an issue in alloy

[QuantumEnigmaa]

Same thing happening on goat (obviously) but currently the error logs on both installations are the following :

ts=2024-11-25T22:28:52.233430109Z level=warn msg="error sending batch, will retry" component=loki.write.default component=client host=write.loki.leopard.p5x2.telekom.de status=500 tenant=leopard error="server returned HTTP status 500 Internal Server Error (500): rpc error: code = DeadlineExceeded desc = context deadline exceeded"

[QuantumEnigmaa]

After restarting both grafana-agents on goat and leopard there are no error logs mentioning any timeouts. Moreover, after checking the timestamps of the previous error logs, I noticed that those errors didn't happen that often and were quite spread in time.

Finally, k8s events can been seen in grafana explore on both installations :

Since this PM is 2 weeks old, maybe some changes that could have happened in the meantime made the agents work again. So I would say on my end that we could close this.

@QuentinBisson any comments ?

[QuentinBisson]

Can you delete the chart and see if it is not a manual change ? As in, me not editing thé CNP by hand 😅

[QuantumEnigmaa]

So I rolled the charts on both installations and I don't see any noticeable change 🤷

[QuentinBisson]

Noice