Cloudwatch showing Insufficient data after changing EKS worker node type

[mrad-bilel]

Envirement

EKS V1.16
Cloudwatch agent version 1.247346.0b249609
we have scaled up our eks worker nodes after that some cloudwatch alarms stack in Insufficient data .

logs from Cloudwatch daemonsets pod

2021/06/04 15:01:11 I! I! Detected the instance is EC2
2021/06/04 15:01:07 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2021/06/04 15:01:07 Reading json config file path: /etc/cwagentconfig/..2021_06_04_15_01_01.806981504/cwagentconfig.json ...
2021/06/04 15:01:07 Find symbolic link /etc/cwagentconfig/..data 
2021/06/04 15:01:07 Find symbolic link /etc/cwagentconfig/cwagentconfig.json 
2021/06/04 15:01:07 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
2021/06/04 15:01:08 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2021/06/04 15:01:09 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/06/04 15:01:10 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/06/04 15:01:11 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/06/04 15:01:11 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
 
2021/06/04 15:01:11 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2021-06-04T15:01:11Z I! Starting AmazonCloudWatchAgent 1.247346.0
2021-06-04T15:01:11Z I! Loaded inputs: cadvisor k8sapiserver
2021-06-04T15:01:11Z I! Loaded aggregators: 
2021-06-04T15:01:11Z I! Loaded processors: ec2tagger k8sdecorator
2021-06-04T15:01:11Z I! Loaded outputs: cloudwatchlogs
2021-06-04T15:01:11Z I! Tags enabled: 
2021-06-04T15:01:11Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-21-173-56.eu-west-3.compute.internal", Flush Interval:1s
2021-06-04T15:01:11Z I! [logagent] starting
2021-06-04T15:01:11Z I! [logagent] found plugin cloudwatchlogs is a log backend
2021-06-04T15:01:11Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2021-06-04T15:01:11Z I! k8sapiserver Switch New Leader: ip-172-21-129-101.eu-west-3.compute.internal
2021-06-04T15:01:11Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2021-06-04T15:01:11Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
2021-06-04T15:01:19Z W! [outputs.cloudwatchlogs] Invalid SequenceToken used, will use new token and retry: The given sequenceToken is invalid. The next expected sequenceToken is: 49618892446055793787834088719197187107160946756573725778
2021-06-04T15:01:19Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 184.77402ms before retrying.
2021-06-04T15:05:18Z I! [processors.ec2tagger] ec2tagger: Refresh is no longer needed, stop refreshTicker.

Any idea please ?

[pingleig]

If your version is 1.247346.0b249609 then you might be missing pod metrics due to docker cgroup driver #171 , can you try the latest version?

btw: If you are also collecting prometheus metrics, due to #209 you might want to use an older version before latest is out. That issue also has links to released manifests.

[mrad-bilel]

Problem resolved by usiung the latest version thanks for your help .