[Bug Fix] Fix Excessive IMDS related error logging #1440
Conversation
| go e.serviceprovider.startServiceProvider() | ||
| // Instance metadata tags is not usable for EKS nodes | ||
| // https://github.com/kubernetes/cloud-provider-aws/issues/762 | ||
| if e.kubernetesMode == "" { |
There was a problem hiding this comment.
This may seem drastic but based on the codebase serviceprovider isn't really used in Kubernetes scenario. It always gets overriden by either Instrumentation or K8sWorkload service name source.
extension/entitystore/ec2Info.go
Outdated
| return | ||
| // Instance metadata tags is not usable for EKS nodes | ||
| // https://github.com/kubernetes/cloud-provider-aws/issues/762 | ||
| if ei.kubernetesMode == "" { |
There was a problem hiding this comment.
we want to keep the retryer on native k8s on ec2 right?
There was a problem hiding this comment.
Fixed in new commit
| // Instance metadata tags is not usable for EKS nodes | ||
| // https://github.com/kubernetes/cloud-provider-aws/issues/762 | ||
| if e.kubernetesMode == "" { | ||
| go e.serviceprovider.startServiceProvider() |
There was a problem hiding this comment.
Does e.serviceprovider.startServiceProvider() do anything beneficial on EKS? Seems strange that we would have to do this just to avoid the IMDS check
There was a problem hiding this comment.
I don't think so. startServiceProvider scrapes IAM role or service name from instance tags so they can be used for service names. In Kubernetes environment neither of these will be used because they have lower priority than Instrumentation or K8sWorkload.
Code wise, I only see serviceprovider public function called in awsentity processor if we are strictly on EC2:
|
Can you provide logs of the built agent on EC2 and EKS? |
| go e.serviceprovider.startServiceProvider() | ||
| // Instance metadata tags is not usable for EKS nodes | ||
| // https://github.com/kubernetes/cloud-provider-aws/issues/762 | ||
| if e.kubernetesMode == "" { |
There was a problem hiding this comment.
similar to the above -- shouldn't this be if e.kubernetesMode != modeEKS ? So that we can account for native k8s?
There was a problem hiding this comment.
I disabled it for all kubernetes environment because serviceprovider isn't used for any Kubernetes related logics. GetMetricServiceNameAndSource is the only public function that is used for awsentity processor and that is only referenced in EC2 specific processor logic:
|
This PR was marked stale due to lack of activity. |
|
pr build failing |
Description of the issue
In version 1.300049.0 and above, the agent will log the above message regardless of log levels on a defined interval. This is because we recently enabled instance tags by default to retrieve ASG name and instance tag name for entity service names. This becomes an issue when instance metadata tags is not enabled which can be majority case since instance metadata tags is an opt-in feature. The issue is especially apparent in EKS since EKS does not support instance metadata tags
Description of changes
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Unit tests
Manual Test
EKS Agent log
Before the fix, we will immediately see EC2MetadataError logs during agent start. In the below logs, we do not see any.
EC2 Agent Journalctl Log
EC2 Agent local log
Requirements
Before commit the code, please do the following steps.
make fmtandmake fmt-shmake lintRelated Issue