The user eks:vpc-resource-controller doesn't have permission to patch cninode

[weijiany]

What happened:

I used the latest vpc-cni(v1.18.1-eksbuild.3) and enabled the pod eni(ENABLE_POD_ENI) in environment variables, then I alwasy received an alerat: {@usr.id:eks:vpc-resource-controller} A Kubernetes user attempted to perform a high number of actions that were denied. I found that there are a lot of error messages in CloudWatch cninodes.vpcresources.k8s.aws \"ip-[ip].[region].compute.internal\" is forbidden: User \"eks:vpc-resource-controller\" cannot patch resource \"cninodes\" in API group \"vpcresources.k8s.aws\" at the cluster scope.

I don't know what happened in EKS cluster, and it caused a lot of noise alerts.

Attach logs

What you expected to happen:

Don't raise this error.

How to reproduce it (as minimally and precisely as possible):

1. Run the following terraform configurations.

locals {
  cluster_name = "test"
}

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "goudan-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-2a", "us-east-2b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = false
}

module "eks" {
  source  = "registry.terraform.io/terraform-aws-modules/eks/aws"
  version = "20.3.0"

  create_iam_role = true

  cluster_name                    = local.cluster_name
  cluster_version                 = "1.29"
  cluster_enabled_log_types       = ["audit", "authenticator"]
  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true

  cluster_addons = {
    "vpc-cni" = {
      addon_version = "v1.18.1-eksbuild.3"
      before_compute           = true
      service_account_role_arn = module.vpc_cni_irsa_role.iam_role_arn
      configuration_values = jsonencode({
        env = {
          ENABLE_POD_ENI                = "true"
          NETWORK_POLICY_ENFORCING_MODE = "strict"
        },
        init = {
          env = {
            DISABLE_TCP_EARLY_DEMUX = "true"
          }
        }
        enableNetworkPolicy = "true"
      })
    }

    "coredns" = {
      resolve_conflicts_on_create = "NONE"
      resolve_conflicts_on_update = "OVERWRITE"

      configuration_values = jsonencode({
        computeType = "Fargate"
      })

      most_recent = true
    }
  }

  create_kms_key            = false
  cluster_encryption_config = {}

  node_security_group_enable_recommended_rules = false
  node_security_group_additional_rules = {
    cluster_to_node_other_ports = {
      description                   = "Cluster API to node by other ports"
      protocol                      = "tcp"
      from_port                     = 1025
      to_port                       = 65535
      type                          = "ingress"
      source_cluster_security_group = true
    }

    node_egress = {
      description = "Egress Freedom"
      cidr_blocks = ["0.0.0.0/0"]
      protocol    = "all"
      from_port   = 0
      to_port     = 65535
      type        = "egress"
    }

    node_to_node_ingress = {
      description = "Node to Node Ingress"
      protocol    = "all"
      from_port   = 0
      to_port     = 65535
      type        = "ingress"
      self        = true
    }
  }

  cluster_security_group_description = "EKS cluster security group."
  cluster_security_group_name        = local.cluster_name

  iam_role_name = local.cluster_name
  iam_role_additional_policies = {
    eks_service = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
  }

  eks_managed_node_groups = {
    test = {
      min_size     = 1
      max_size     = 1
      desired_size = 1

      instance_types = ["c5.large"]
      capacity_type  = "SPOT"
    }
  }

  fargate_profiles = {
    coredns = {
      name = "coredns"

      selectors = [
        {
          namespace = "kube-system"
          labels = {
            "k8s-app" : "kube-dns"
          }
        }
      ]

      subnet_ids = module.vpc.private_subnets

      iam_role_additional_policies = {
        cloudwatch = "arn:aws:iam::aws:policy/AWSOpsWorksCloudWatchLogs"
      }

      tags = {
        Owner = "coredns"
      }
    }
  }

  enable_cluster_creator_admin_permissions = true
}

module "vpc_cni_irsa_role" {
  source    = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version   = ">= 5.34.0, < 6.0.0"
  role_name = "${local.cluster_name}-cluster-cni"

  attach_vpc_cni_policy = true
  vpc_cni_enable_ipv4   = true

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-node"]
    }
  }
}

resource "aws_security_group_rule" "lb_to_node_group" {
  type              = "ingress"
  from_port         = 53
  to_port           = 53
  protocol          = "tcp"
  security_group_id = module.eks.node_security_group_id
  cidr_blocks       = [module.vpc.vpc_cidr_block]
}


resource "aws_security_group_rule" "node_group_to_cluster_primary_tcp_53" {
  type                     = "ingress"
  from_port                = 53
  to_port                  = 53
  protocol                 = "tcp"
  security_group_id        = module.eks.cluster_primary_security_group_id
  source_security_group_id = module.eks.node_security_group_id
}

resource "aws_security_group_rule" "node_group_to_cluster_primary_udp_53" {
  type                     = "ingress"
  from_port                = 53
  to_port                  = 53
  protocol                 = "udp"
  security_group_id        = module.eks.cluster_primary_security_group_id
  source_security_group_id = module.eks.node_security_group_id
}

resource "aws_security_group_rule" "node_group_to_cluster_primary_tcp_9153" {
  type                     = "ingress"
  from_port                = 9153
  to_port                  = 9153
  protocol                 = "tcp"
  security_group_id        = module.eks.cluster_primary_security_group_id
  source_security_group_id = module.eks.node_security_group_id
  description              = "Allow the CoreDNS metric endpoint on port 9153 to be accessible by all pods"
}

2. Open AWS Cloud Watch, search: "eks:vpc-resource-controller" patch cninode, you will find the 403 Failure.

Anything else we need to know?:

Environment:

• Kubernetes version: v1.29.4-eks-036c24b
• CNI Version: v1.18.1-eksbuild.3
• AMI Id: ami-07bf4230c9e443622

[sushrk]

@weijiany thanks for the report. The fix from EKS on 1.29 clusters is in progress for this issue.
Meanwhile as a workaround, could you try adding the missing permission for CNINode CRD in the vpc-resource-controller-role cluster role?

You can apply this manifest file in your cluster to add the missing permissions and required fields in the CRD on 1.29:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.9.0
  creationTimestamp: null
  name: cninodes.vpcresources.k8s.aws
spec:
  group: vpcresources.k8s.aws
  names:
    kind: CNINode
    listKind: CNINodeList
    plural: cninodes
    shortNames:
    - cnd
    singular: cninode
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - description: The features delegated to VPC resource controller
      jsonPath: .spec.features
      name: Features
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: 'Important: Run "make" to regenerate code after modifying
              this file CNINodeSpec defines the desired state of CNINode'
            properties:
              features:
                items:
                  description: Feature is a type of feature being supported by VPC
                    resource controller and other AWS Services
                  properties:
                    name:
                      description: FeatureName is a type of feature name supported
                        by AWS VPC CNI. It can be Security Group for Pods, custom
                        networking, or others
                      type: string
                    value:
                      type: string
                  type: object
                type: array
              tags:
                additionalProperties:
                  type: string
                description: Additional tag key/value added to all network interfaces
                  provisioned by the vpc-resource-controller and VPC-CNI
                type: object
            type: object
          status:
            description: CNINodeStatus defines the managed VPC resources.
            type: object
        type: object
    served: true
    storage: true
    subresources: {}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vpc-resource-controller-role
rules:
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - update
      - patch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - get
      - patch
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - patch
      - watch
  - apiGroups:
      - ""
    resources:
      - serviceaccounts
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - crd.k8s.amazonaws.com
    resources:
      - eniconfigs
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - vpcresources.k8s.aws
    resources:
      - securitygrouppolicies
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - vpcresources.k8s.aws
    resources:
      - cninodes
    verbs:
      - create
      - get
      - list
      - patch
      - update
      - watch

[weijiany]

Hi @sushrk thanks for your reply, I saw that the latest EKS version is 1.30, does it be fixed in this version?

[weijiany]

I upgrade EKS to 1.30, this alert doesn't appear anymore. It looks like it has been resolved in 1.30, I will close this issue. Thanks @sushrk help. 😃

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

[sushrk]

Thank you. This issue has been resolved across all available EKS versions.