Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ENV var RUN_IN_AWS to be passed to container #50

Open
Bourne-ID opened this issue Apr 1, 2024 · 0 comments
Open

Allow ENV var RUN_IN_AWS to be passed to container #50

Bourne-ID opened this issue Apr 1, 2024 · 0 comments

Comments

@Bourne-ID
Copy link
Contributor

We have a race condition where the container insights pod starts before the full VPC CNI networking is online. This can cause a connection failure to the EC2 Metadata endpoint, which reverts the container into OnPremise mode. For context we use IRSA for AWS permissions in EKS clusters.

When in OnPremise mode, the container looks for credentials stored on the local file system, and fails to load them

E! [EC2] Cannot get EC2 Metadata from IMDS: EC2 metadata is not available.
...
I! Detected the instance is OnPremise
...
E! [outputs.cloudwatchlogs] Aws error received when sending logs to /aws/containerinsights/<cluster>/performance/<node>>: SharedCredsLoad: failed to load shared credentials file
caused by: FailedRead: unable to open file
caused by: open /root/.aws/credentials: no such file or directory

This causes the agent to fail to publish metrics to the Log Group, causing a loss of visibility in CloudZero.

Setting the ENV Variable RUN_IN_AWS allows the check to bypass IMDS and default to using the IRSA permissions on the Service Account.

https://github.com/Cloudzero/amazon-cloudwatch-agent/blob/cloudzero-optimized/translator/util/sdkutil.go#L36

However, the helm chart is not configured to define this ENV var, or custom ENVs.

I see two acceptances here:

  1. This specific ENV is a variable in the values.yaml and passed into the Daemonset (similar to https://github.com/Cloudzero/cloudzero-k8s-charts/blob/develop/stable/cloudzero-cloudwatch-metrics/templates/daemonset.yaml#L56)
  2. Custom ENVs can be defined.

I personally prefer 2 for future/custom cases, but open to 1 for more user friendly choices on values.yaml. I'll likely make 2 PRs with both solutions for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant