You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a race condition where the container insights pod starts before the full VPC CNI networking is online. This can cause a connection failure to the EC2 Metadata endpoint, which reverts the container into OnPremise mode. For context we use IRSA for AWS permissions in EKS clusters.
When in OnPremise mode, the container looks for credentials stored on the local file system, and fails to load them
E! [EC2] Cannot get EC2 Metadata from IMDS: EC2 metadata is not available.
...
I! Detected the instance is OnPremise
...
E! [outputs.cloudwatchlogs] Aws error received when sending logs to /aws/containerinsights/<cluster>/performance/<node>>: SharedCredsLoad: failed to load shared credentials file
caused by: FailedRead: unable to open file
caused by: open /root/.aws/credentials: no such file or directory
This causes the agent to fail to publish metrics to the Log Group, causing a loss of visibility in CloudZero.
Setting the ENV Variable RUN_IN_AWS allows the check to bypass IMDS and default to using the IRSA permissions on the Service Account.
I personally prefer 2 for future/custom cases, but open to 1 for more user friendly choices on values.yaml. I'll likely make 2 PRs with both solutions for review.
The text was updated successfully, but these errors were encountered:
We have a race condition where the container insights pod starts before the full VPC CNI networking is online. This can cause a connection failure to the EC2 Metadata endpoint, which reverts the container into OnPremise mode. For context we use IRSA for AWS permissions in EKS clusters.
When in OnPremise mode, the container looks for credentials stored on the local file system, and fails to load them
This causes the agent to fail to publish metrics to the Log Group, causing a loss of visibility in CloudZero.
Setting the ENV Variable RUN_IN_AWS allows the check to bypass IMDS and default to using the IRSA permissions on the Service Account.
https://github.com/Cloudzero/amazon-cloudwatch-agent/blob/cloudzero-optimized/translator/util/sdkutil.go#L36
However, the helm chart is not configured to define this ENV var, or custom ENVs.
I see two acceptances here:
I personally prefer 2 for future/custom cases, but open to 1 for more user friendly choices on values.yaml. I'll likely make 2 PRs with both solutions for review.
The text was updated successfully, but these errors were encountered: