eks pod taking long time to configure aws credentials by imds

[prasanjeet-philips]

Describe the bug
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-commandline-fleet.html#install-CloudWatch-Agent-iam_user-first

I am running a docker container on eks managed node
I am seeing the it is taking exactly 8 mins to configure aws and creating a pipe to the loggroup
I am using amazonlinx container

Dockerfile

# Use Amazon Linux 2 as the base image
FROM amazonlinux:2

# Install systemd and necessary tools
RUN yum -y update && \
    yum -y install systemd sudo unzip nano amazon-cloudwatch-agent jq && \
    yum clean all

# Install AWS CLI v2
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
    unzip awscliv2.zip && \
    ./aws/install && \
    rm -rf awscliv2.zip aws

# Copy CloudWatch Agent Configuration
COPY cloudwatch-agent-config.json /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json


# Create Logs Directory and Initialize a Fresh app.log
RUN mkdir -p /var/log/my-app && \
    touch /var/log/my-app/application.log && \
    echo "Log initialized: $(date)" >> /var/log/my-app/application.log

# # Copy custom scripts
COPY ./src/configure-cw.sh /root/configure-cw.sh
COPY ./src/main.sh /root/main.sh
COPY ./src/startup.sh /root/startup.sh

# Ensure all custom scripts are executable
RUN chmod +x /root/configure-cw.sh /root/main.sh /root/startup.sh

# Set working directory
WORKDIR /root

startup.sh

#!/bin/sh

exec /opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent

This is the logs from the cloudwatch log file inside pod

`sh-4.2# cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

2024/12/05 05:09:33 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2024/12/05 05:09:33 D! config [agent]
  collection_jitter = "0s"
  debug = false
  flush_interval = "1s"
  flush_jitter = "0s"
  hostname = ""
  interval = "60s"
  logfile = "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  logtarget = "lumberjack"
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  omit_hostname = false
  precision = ""
  quiet = false
  round_interval = false

[inputs]

  [[inputs.logfile]]
    destination = "cloudwatchlogs"
    file_state_folder = "/opt/aws/amazon-cloudwatch-agent/logs/state"

    [[inputs.logfile.file_config]]
      file_path = "/var/log/my-app/application.log"
      from_beginning = true
      log_group_class = "STANDARD"
      log_group_name = "provisioning-lg"
      log_stream_name = "provisioning-lg-provision"
      multi_line_start_pattern = "^\\d{2}-\\d{2}-\\d{4} \\d{2}:\\d{2}:\\d{2} (AM|PM)"
      pipe = false
      retention_in_days = 7
      timestamp_layout = ["_2-01-2006 03-04-05 PM", "_2-1-2006 03-04-05 PM"]
      timestamp_regex = "(\\d{1,2}-\\s{0,1}\\d{1,2}-\\d{4} \\d{2}-\\d{2}-\\d{2} \\w{2})"
      timezone = "UTC"

[outputs]

  [[outputs.cloudwatchlogs]]
    force_flush_interval = "5s"
    log_stream_name = "auto"
    mode = "K8E"
    region = "eu-west-1"
    region_type = "ACJ"
2024/12/05 05:09:33 I! Config has been translated into YAML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.yaml
2024/12/05 05:09:33 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2024/12/05 05:09:33 I! Valid Json input schema.
2024/12/05 05:09:33 I! Detected runAsUser: root
2024/12/05 05:09:33 I! Changing ownership of [/opt/aws/amazon-cloudwatch-agent/logs /opt/aws/amazon-cloudwatch-agent/etc /opt/aws/amazon-cloudwatch-agent/var] to 0:0
2024-12-05T05:09:33Z I! Starting AmazonCloudWatchAgent CWAgent/1.300044.0 (go1.22.5; linux; amd64) with log file /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log with log target lumberjack
2024-12-05T05:09:33Z I! AWS SDK log level not set
2024-12-05T05:09:33Z I! creating new logs agent
2024-12-05T05:09:33Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"test-job-mtq7c", Flush Interval:1s
2024-12-05T05:09:33Z I! [logagent] start logs plugin file paths [/var/log/my-app/application.log]
2024-12-05T05:09:33Z I! [inputs.logfile] turned on logs plugin
2024-12-05T05:09:33Z I! [logagent] starting
2024-12-05T05:09:33Z I! [logagent] found plugin cloudwatchlogs is a log backend
2024-12-05T05:09:33Z I! [logagent] found plugin logfile is a log collection
2024-12-05T05:09:33Z I! [logagent] start logs plugin file paths [/var/log/my-app/application.log]
2024-12-05T05:09:33Z I! [inputs.logfile] turned on logs plugin
2024-12-05T05:09:34Z I! First time setting retention for log group provisioning-lg, update map to avoid setting twice
2024-12-05T05:17:35Z I! [outputs.cloudwatchlogs] Configured middleware on AWS client
2024-12-05T05:17:35Z I! [logagent] piping log from provisioning-lg/provisioning-lg-provision(/var/log/my-app/application.log) to cloudwatchlogs with retention 7`

Steps to reproduce
kubectl logs test-job-k9njx

D! [EC2] Found active network interface
I! imds retry client will retry 1 timesD! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! could not get hostname without imds v1 fallback enable thus enable fallback
D! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! should retry true for imds error : RequestError: send request failed
caused by: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers)D! could not get instance document without imds v1 fallback enable thus enable fallback
I! Detected the instance is EC2
2024/12/05 04:39:27 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2024/12/05 04:39:27 I! Valid Json input schema.
I! Detecting run_as_user...
2024/12/05 04:39:27 Configuration validation first phase succeeded
I! Detecting run_as_user...

What did you expect to see?
It should configure aws within seconds

expected:

2024-12-04T14:41:32Z I! First time setting retention for log group provisioningv7-lg, update map to avoid setting twice
2024-12-04T14:41:32Z I! [outputs.cloudwatchlogs] Configured middleware on AWS client
2024-12-04T14:41:32Z I! [logagent] piping log from provisioningv7-lg/provisioningv7-lg-stream(/var/log/my-app/application.log) to cloudwatchlogs with retention 7

What did you see instead?
but here it is taking 8 mins

actual:

2024-12-05T05:09:34Z I! First time setting retention for log group provisioning-lg, update map to avoid setting twice
2024-12-05T05:17:35Z I! [outputs.cloudwatchlogs] Configured middleware on AWS client

What version did you use?

 cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"

What config did you use?
cloudwatch-agent-config.json

{
  "agent": {
    "region": "eu-west-1"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/my-app/application.log",
            "log_group_name": "provisioning-lg",
            "log_stream_name": "provisioning-lg-provision",
            "timezone": "UTC",
            "multi_line_start_pattern": "^\\d{2}-\\d{2}-\\d{4} \\d{2}:\\d{2}:\\d{2} (AM|PM)",
            "log_group_class": "STANDARD",
            "timestamp_format": "%d-%m-%Y %I-%M-%S %p",
            "retention_in_days": 7
          }
        ]
      }
    },
    "log_stream_name": "auto"
  }
}

Environment
Amazon Linux:2

Additional context
Please suggest. I need to work on this item.

Observation it takes exactly 8 mins to configure aws sdk