[k8s] Pod metrics is gone when using containerd as runtime #188

pingleig · 2021-03-21T01:09:45Z

This is exported from internal ticket

TL;DR

The latest image is released, if you were using temp image from this comment #188 (comment) please update to the latest tag.

If the error message W! No pod metric collected, metrics count is still 7 is containerd socket mounted? https://github.com/aws/amazon-cloudwatch-agent/issues/188leads you to this issue

make sure you have updated the yaml to mount the containerd socket into the cloudwatch-agent pod
the path for containerd socket may not be in the standard location. e.g. bottlerocket uses /run/dockershim.sock instead of /run/containerd/containerd.sock

Background

We were relying on pause container to have POD for detecting pod, which is the case for docker but not for containerd containerd/cri#922 (comment)

User will not see pod metrics in container insight dashboard and they will find the following log which is introduced in #171

amazon-cloudwatch-agent/plugins/inputs/cadvisor/container_info_processor.go

Line 72 in fbdd619

log.Printf("W! No pod metric collected, metrics count is still %d", beforePod)

The root cause is we are expecting containerName == 'POD' to mark a path as pod

amazon-cloudwatch-agent/plugins/inputs/cadvisor/container_info_processor.go

Lines 119 to 126 in fbdd619

    
           if containerName != infraContainerName { 
        
           	tags[ContainerNamekey] = containerName 
        
           	tags[ContainerIdkey] = path.Base(info.Name) 
        
           	containerType = TypeContainer 
        
           } else { 
        
           	// NOTE: the pod here is only used by NetMetricExtractor, 
        
           	// other pod info like CPU, Mem are dealt within in processPod. 
        
           	containerType = TypePod

Fix

code should be similar to Fix pod metric filtering for containerd DataDog/integrations-core#2283
manifest need to mount containerd sock into cwagent daemonset container

Release

The fix will be included in next release, the release date is not determined (yet).

The text was updated successfully, but these errors were encountered:

pingleig · 2021-03-22T05:10:34Z

Created a temp image based on #189 ~~public.ecr.aws/p5m3p1a7/cwagent-k8s-containerd-pod:0.1~~ (the latest official release now contains this fix) and the daemonset yaml need to be udpated to mount /run/containerd/containerd.sock

NOTE: If you are using bottlerocket on eks, the socket on host is different due to bottlerocket-os/bottlerocket@91810c8 You need to (and only need to) replace the volumes part to pick the right sock on host. (Full snippet is at end of comment).

      volumes:
       # ... 
        - name: containerdsock
          hostPath:
            # path: /run/containerd/containerd.sock
            # bottlerocket does not mount containerd sock at normal place
            # https://github.com/bottlerocket-os/bottlerocket/commit/91810c85b83ff4c3660b496e243ef8b55df0973b
            path: /run/dockershim.sock

Default containerd path

When host (and kubelet) is using /run/containerd/containerd.sock

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cloudwatch-agent
  namespace: amazon-cloudwatch
spec:
  selector:
    matchLabels:
      name: cloudwatch-agent
  template:
    metadata:
      labels:
        name: cloudwatch-agent
    spec:
      containers:
        - name: cloudwatch-agent
          image: public.ecr.aws/p5m3p1a7/cwagent-k8s-containerd-pod:0.1
          imagePullPolicy: Always
          #ports:
          #  - containerPort: 8125
          #    hostPort: 8125
          #    protocol: UDP
          resources:
            limits:
              cpu: 200m
              memory: 200Mi
            requests:
              cpu: 200m
              memory: 200Mi
          # Please don't change below envs
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: K8S_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: CI_VERSION
              value: "k8s/1.3.0"
          # Please don't change the mountPath
          volumeMounts:
            - name: cwagentconfig
              mountPath: /etc/cwagentconfig
            - name: rootfs
              mountPath: /rootfs
              readOnly: true
            - name: dockersock
              mountPath: /var/run/docker.sock
              readOnly: true
            - name: varlibdocker
              mountPath: /var/lib/docker
              readOnly: true
            - name: containerdsock
              mountPath: /run/containerd/containerd.sock
              readOnly: true
            - name: sys
              mountPath: /sys
              readOnly: true
            - name: devdisk
              mountPath: /dev/disk
              readOnly: true
      volumes:
        - name: cwagentconfig
          configMap:
            name: cwagentconfig
        - name: rootfs
          hostPath:
            path: /
        - name: dockersock
          hostPath:
            path: /var/run/docker.sock
        - name: varlibdocker
          hostPath:
            path: /var/lib/docker
        - name: containerdsock
          hostPath:
            path: /run/containerd/containerd.sock
        - name: sys
          hostPath:
            path: /sys
        - name: devdisk
          hostPath:
            path: /dev/disk/
      terminationGracePeriodSeconds: 60
      serviceAccountName: cloudwatch-agent

Non default containerd path

NOTE: You only need to change the volumes, when mount into cloudwatch agent container, you should still put it at default path.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cloudwatch-agent
  namespace: amazon-cloudwatch
spec:
  selector:
    matchLabels:
      name: cloudwatch-agent
  template:
    metadata:
      labels:
        name: cloudwatch-agent
    spec:
      # aws eks update-kubeconfig --name eks-pod-metric-missing --region us-west-2
      containers:
        - name: cloudwatch-agent
          image: public.ecr.aws/p5m3p1a7/cwagent-k8s-containerd-pod:0.1
          imagePullPolicy: Always
          #ports:
          #  - containerPort: 8125
          #    hostPort: 8125
          #    protocol: UDP
          resources:
            limits:
              cpu: 200m
              memory: 200Mi
            requests:
              cpu: 200m
              memory: 200Mi
          # Please don't change below envs
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: K8S_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: CI_VERSION
              value: "k8s/1.3.0"
          # Please don't change the mountPath
          volumeMounts:
            - name: cwagentconfig
              mountPath: /etc/cwagentconfig
            - name: rootfs
              mountPath: /rootfs
              readOnly: true
            - name: dockersock
              mountPath: /var/run/docker.sock
              readOnly: true
            - name: varlibdocker
              mountPath: /var/lib/docker
              readOnly: true
            - name: containerdsock
              mountPath: /run/containerd/containerd.sock
              readOnly: true
            - name: sys
              mountPath: /sys
              readOnly: true
            - name: devdisk
              mountPath: /dev/disk
              readOnly: true
      volumes:
        - name: cwagentconfig
          configMap:
            name: cwagentconfig
        - name: rootfs
          hostPath:
            path: /
        - name: dockersock
          hostPath:
            path: /var/run/docker.sock
        - name: varlibdocker
          hostPath:
            path: /var/lib/docker
        - name: containerdsock
          hostPath:
            # path: /run/containerd/containerd.sock
            # bottle rocket does not mount containerd sock at normal place
            # https://github.com/bottlerocket-os/bottlerocket/commit/91810c85b83ff4c3660b496e243ef8b55df0973b
            path: /run/dockershim.sock
        - name: sys
          hostPath:
            path: /sys
        - name: devdisk
          hostPath:
            path: /dev/disk/
      terminationGracePeriodSeconds: 60
      serviceAccountName: cloudwatch-agent

pingleig · 2021-03-22T21:06:56Z

Another known issue is because we are using cadvisor, pod level filesystem usage is ignored

    "container_filesystem_available",
    "container_filesystem_capacity",
    "container_filesystem_usage",
    "container_filesystem_utilization"

https://github.com/google/cadvisor/blob/291c215c5ddc5216659b5e793a98a0ba9f104afb/container/containerd/handler.go#L163-L167

func (h *containerdContainerHandler) GetSpec() (info.ContainerSpec, error) {
	// TODO: Since we dont collect disk usage stats for containerd, we set hasFilesystem
	// to false. Revisit when we support disk usage stats for containerd
	hasFilesystem := false
	spec, err := common.GetSpec(h.cgroupPaths, h.machineInfoFactory, h.needNet(), hasFilesystem)
	spec.Labels = h.labels
	spec.Envs = h.envs
	spec.Image = h.image

	return spec, err
}

pingleig · 2021-03-23T01:37:09Z

NOTE: container file system usage is not provided after switching to containerd google/cadvisor#2785

Created another issue to track the container filesystem metrics #192

pingleig · 2021-03-31T00:47:13Z

Reopen this issue since we are still in the release process, and the official container insight public doc plus sample manifest is not updated yet.

pingleig · 2021-04-23T17:07:41Z

Close since the release is out

fitchtech · 2021-08-20T23:34:21Z

This needs fixed within the official helm charts for EKS
https://github.com/aws/eks-charts/blob/master/stable/aws-cloudwatch-metrics/templates/daemonset.yaml

fitchtech · 2021-08-21T00:11:32Z

@pingleig I have tried applying the fix listed above exactly as is on EKS with the containerd runtime enabled. However, I'm still getting the same error messages:

2021-08-21T00:08:59Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2021-08-21T00:08:59Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
2021-08-21T00:09:00Z W! No pod metric collected, metrics count is still 5 is containerd socket mounted? #188
2021-08-21T00:09:05Z W! [outputs.cloudwatchlogs] Invalid SequenceToken used, will use new token and retry: The given sequenceToken is invalid. The next expected sequenceToken is: 49605661750447750614958043896578931231172344896032866930
2021-08-21T00:09:05Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 105.761168ms before retrying.

Support for containerd runtime on EKS was added in July when EKS 1.21 was released.
https://aws.amazon.com/blogs/containers/amazon-eks-1-21-released/

pingleig · 2021-08-21T05:21:05Z

@fitchtech. The containerd socket on host is in a different path (same as bottlerocket). This is PR for EKS AMI https://github.com/awslabs/amazon-eks-ami/pull/698/files and the config file https://github.com/awslabs/amazon-eks-ami/blob/8450297eb2ef87fe5cbbce52a86ddcdc8b2e6716/files/containerd-config.toml#L1-L6

[grpc]
address = "/run/dockershim.sock"

You can follow non default path in #188 (comment)

          hostPath:
            # path: /run/containerd/containerd.sock
            # bottle rocket does not mount containerd sock at normal place
            # https://github.com/bottlerocket-os/bottlerocket/commit/91810c85b83ff4c3660b496e243ef8b55df0973b
            path: /run/dockershim.sock

cc @sethAmazon since both EKS EC2 and Bottlerocket are using /run/dockershim.sock we may change this to =default. Though I was testing using kops at that time, which uses /run/containerd/containerd.sock. I am not sure if it's possible to have one manifest that works for both in our example manifest. Though it should doable for helm.

fitchtech · 2021-08-21T07:46:49Z

@pingleig that worked, thank you. One additional change I had to make is to enable hostNetwork, cause the EC2 instances in my EKS 1.21 node group has the Instance MetaData Service (IMDS) restricted per the EKS security best practices . You have to set hostNetwork: true for it to be able to start up. Once I did everything loaded in the ContainerInsights console.

With hostNetwork: false I get this

2021/08/21 07:23:59 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2021-08-21T07:23:59Z I! Starting AmazonCloudWatchAgent 1.247349.0
2021-08-21T07:23:59Z I! Loaded inputs: k8sapiserver cadvisor
2021-08-21T07:23:59Z I! Loaded aggregators: 
2021-08-21T07:23:59Z I! Loaded processors: ec2tagger k8sdecorator
2021-08-21T07:23:59Z I! Loaded outputs: cloudwatchlogs
2021-08-21T07:23:59Z I! Tags enabled: 
2021-08-21T07:23:59Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-10-106-12-9.ec2.internal", Flush Interval:1s
2021-08-21T07:23:59Z I! [logagent] starting
2021-08-21T07:23:59Z I! [logagent] found plugin cloudwatchlogs is a log backend

With hostNetwork: true

2021/08/21 07:28:18 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2021-08-21T07:28:18Z I! Starting AmazonCloudWatchAgent 1.247349.0
2021-08-21T07:28:18Z I! Loaded inputs: cadvisor k8sapiserver
2021-08-21T07:28:18Z I! Loaded aggregators: 
2021-08-21T07:28:18Z I! Loaded processors: ec2tagger k8sdecorator
2021-08-21T07:28:18Z I! Loaded outputs: cloudwatchlogs
2021-08-21T07:28:18Z I! Tags enabled: 
2021-08-21T07:28:18Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-10-106-12-9.ec2.internal", Flush Interval:1s
2021-08-21T07:28:18Z I! [logagent] starting
2021-08-21T07:28:18Z I! [logagent] found plugin cloudwatchlogs is a log backend
2021-08-21T07:28:18Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2021-08-21T07:28:18Z I! k8sapiserver Switch New Leader: ip-10-106-12-14.ec2.internal
2021-08-21T07:28:19Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2021-08-21T07:28:19Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes
2021-08-21T07:28:26Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 137.608142ms before retrying.
2021-08-21T07:33:34Z I! [processors.ec2tagger] ec2tagger: Refresh is no longer needed, stop refreshTicker.

ec2tagger doesn't like not being able to access the instance metadata service and the containers will restart. Once I set hostNetwork to true I started seeing metrics flow into ContainerInsights. This was even though the DaemonSet is set to a service account that using IAM Roles for Service Accounts (IRSA) with a policy that give it ec2:DescribeVolumes & ec2:DescribeTags

Can an update be made that allows this to work without host network enabled on the daemonset?

fitchtech · 2021-08-21T07:58:50Z

Also, the IAM policy document attached to the IRSA role needs allow sts:AssumeRoleWithWebIdentity & sts:AssumeRole resource restricted to the IRSA role ARN or it will throw access denied errors and assume role API call.

fitchtech · 2021-08-21T08:19:55Z

@fitchtech. The containerd socket on host is in a different path (same as bottlerocket). This is PR for EKS AMI https://github.com/awslabs/amazon-eks-ami/pull/698/files and the config file https://github.com/awslabs/amazon-eks-ami/blob/8450297eb2ef87fe5cbbce52a86ddcdc8b2e6716/files/containerd-config.toml#L1-L6
[grpc]
address = "/run/dockershim.sock"
You can follow non default path in #188 (comment)
          hostPath:
            # path: /run/containerd/containerd.sock
            # bottle rocket does not mount containerd sock at normal place
            # https://github.com/bottlerocket-os/bottlerocket/commit/91810c85b83ff4c3660b496e243ef8b55df0973b
            path: /run/dockershim.sock
cc @sethAmazon since both EKS EC2 and Bottlerocket are using /run/dockershim.sock we may change this to =default. Though I was testing using kops at that time, which uses /run/containerd/containerd.sock. I am not sure if it's possible to have one manifest that works for both in our example manifest. Though it should doable for helm.

The official EKS helm charts for CloudWatch Metrics should be updated to do this instead of applying manifests so that you can use helm templates to conditionally set those based on values provided.

aws/amazon-cloudwatch-agent#188

pingleig added bug Something isn't working component/container-insight area/k8s Kubernetes labels Mar 21, 2021

pingleig self-assigned this Mar 21, 2021

pingleig added a commit to pingleig/amazon-cloudwatch-agent that referenced this issue Mar 22, 2021

cadvisor: Fix pod detection for containerd aws#188

8784c3c

pingleig mentioned this issue Mar 22, 2021

cadvisor: Fix pod detection for containerd runtime on k8s #189

Merged

6 tasks

pingleig mentioned this issue Mar 23, 2021

[k8s][containerd] container file system metrics is not supported by cadvisor for contianerd #192

Open

pingleig added this to the v1.247347.6 milestone Mar 23, 2021

pingleig closed this as completed in #189 Mar 24, 2021

pingleig added a commit that referenced this issue Mar 24, 2021

cadvisor: Fix pod detection for containerd #188

774b4ce

pingleig reopened this Mar 31, 2021

This was referenced Apr 22, 2021

daemonset: Add containerd socket mount aws-samples/amazon-cloudwatch-container-insights#72

Merged

Release 1.3.6 aws-samples/amazon-cloudwatch-container-insights#73

Merged

pingleig closed this as completed Apr 23, 2021

pingleig mentioned this issue May 18, 2021

[prometheus] agent is printing all the scraped metrics in log #209

Closed

fitchtech mentioned this issue Aug 20, 2021

CloudWatch Metrics Fails on EKS with ContainerD Runtime aws/eks-charts#582

Open

fitchtech mentioned this issue Aug 21, 2021

CloudWatch Agent Fails on EKS with ContainerD Runtime #261

Closed

pingleig reopened this Aug 21, 2021

pingleig pinned this issue Aug 21, 2021

fitchtech mentioned this issue Nov 12, 2021

Add support for EKS ContainerD Runtime in CloudWatch Metrics charts aws/eks-charts#638

Open

SpringMT mentioned this issue Mar 20, 2022

Do not collect pod metrics using Bottlerocket aws-observability/aws-otel-collector#1071

Closed

bersanf mentioned this issue Jun 30, 2022

[EKS] [request]: Better support for removing instance metadata endpoint access aws/containers-roadmap#1060

Open

kiamran pushed a commit to pileus-cloud/charts that referenced this issue Dec 27, 2022

add mount of /run/containerd/containerd.sock

d30ed76

aws/amazon-cloudwatch-agent#188

sky333999 pushed a commit that referenced this issue Jul 21, 2023

Deprecate old metric decorator code (#188)

0247587

zhihonl mentioned this issue Sep 26, 2023

CCWA 1.300026.3 Known Issues zhihonl/amazon-cloudwatch-agent#5

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[k8s] Pod metrics is gone when using containerd as runtime #188

[k8s] Pod metrics is gone when using containerd as runtime #188

pingleig commented Mar 21, 2021 •

edited

Loading

pingleig commented Mar 22, 2021 •

edited

Loading

pingleig commented Mar 22, 2021

pingleig commented Mar 23, 2021 •

edited

Loading

pingleig commented Mar 31, 2021 •

edited

Loading

pingleig commented Apr 23, 2021

fitchtech commented Aug 20, 2021

fitchtech commented Aug 21, 2021 •

edited

Loading

pingleig commented Aug 21, 2021

fitchtech commented Aug 21, 2021 •

edited

Loading

fitchtech commented Aug 21, 2021

fitchtech commented Aug 21, 2021

[k8s] Pod metrics is gone when using containerd as runtime #188

[k8s] Pod metrics is gone when using containerd as runtime #188

Comments

pingleig commented Mar 21, 2021 • edited Loading

TL;DR

Background

Fix

Release

pingleig commented Mar 22, 2021 • edited Loading

Default containerd path

Non default containerd path

pingleig commented Mar 22, 2021

pingleig commented Mar 23, 2021 • edited Loading

pingleig commented Mar 31, 2021 • edited Loading

pingleig commented Apr 23, 2021

fitchtech commented Aug 20, 2021

fitchtech commented Aug 21, 2021 • edited Loading

pingleig commented Aug 21, 2021

fitchtech commented Aug 21, 2021 • edited Loading

fitchtech commented Aug 21, 2021

fitchtech commented Aug 21, 2021

pingleig commented Mar 21, 2021 •

edited

Loading

pingleig commented Mar 22, 2021 •

edited

Loading

pingleig commented Mar 23, 2021 •

edited

Loading

pingleig commented Mar 31, 2021 •

edited

Loading

fitchtech commented Aug 21, 2021 •

edited

Loading

fitchtech commented Aug 21, 2021 •

edited

Loading