Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when using EKS Pod Identity as credentials provider #103

Open
Bart-Leboeuf opened this issue Dec 7, 2023 · 4 comments
Open

Error when using EKS Pod Identity as credentials provider #103

Bart-Leboeuf opened this issue Dec 7, 2023 · 4 comments
Assignees
@vishal-chdhry

Comments

@Bart-Leboeuf
Copy link

We've tried using EKS Pod Identity with kyervno-notation-aws (V1). It seems that the authentication endpoint is not taken into account by the application, so the role is not assumed. Using IRSA, it works correctly. In the same cluster, we have other containers that work perfectly with EKS Pod Identity.

Using IRSA :

2023-12-07T15:31:54.204Z	INFO	verifier/client.go:175	Token is authorized {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name: GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[{Manager:kyverno-notation-aws Operation:Update APIVersion:authentication.k8s.io/v1 Time:2023-12-07 15:31:54 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:spec":{"f:token":{}}} Subresource:}]} Spec:{Token:eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ5ODk4ZmUzYzQ1YmIyMWRhMjM5MzkyZGIxOTI0ZmUzYzVjNGQ2OWQifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTczMzQ5NjQ2NSwiaWF0IjoxNzAxOTYwNDY1LCJpc3MiOiJodHRwczovL29pZGMuZWtzLmV1LXdlc3QtMy5hbWF6b25hd3MuY29tL2lkLzcwN0Q4NDc0OTREMUQ0ODA4QjI4RDYzQTJCRDBEQUY3Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJreXZlcm5vIiwicG9kIjp7Im5hbWUiOiJreXZlcm5vLXJlcG9ydHMtY29udHJvbGxlci1kOGI3ZDc0OTgtcXA4a24iLCJ1aWQiOiJhNzEyNmY3OC04ZWVkLTRkMGUtOWU2ZS05OTAyY2Q2NzI4ZTAifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6Imt5dmVybm8tcmVwb3J0cy1jb250cm9sbGVyIiwidWlkIjoiNTg4ODY1OTItYzI5Ni00MmY4LWI0N2YtOGVjOWFlNWViZDZjIn0sIndhcm5hZnRlciI6MTcwMTk2NDA3Mn0sIm5iZiI6MTcwMTk2MDQ2NSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt5dmVybm86a3l2ZXJuby1yZXBvcnRzLWNvbnRyb2xsZXIifQ.hFsAk9fKz60IOM7UxeCcIlUHEecbBZzg54LZW73wsUegutpynQujTUmS1wg3tcGKMIP6MMO_SLHJjVExaXXkKxhA-jIjGbX7bQz2mamuSaR171cmyWQOZ7XQTvu5D34Dw39836DGHOBv7BJqT_e6BTz31hnbk-N9ZJboj27MkBH9rRanbtdYqOLv40x6bIz41kUuRHH6OqfvAZ7_dk14bDbdM6X5srSwPm9P2oO2ojOl3hBKMtq7dXU_k5-WHYniPbXXix5wMJDgEyaxw1PCElD8AnG5ZwlksDkt-hKneGV5vIABIjF5Fqk96lrWcR8_2SUGR5g4h8y_DACEfSpctA Audiences:[]} Status:{Authenticated:true User:{Username:system:serviceaccount:kyverno:kyverno-reports-controller UID:58886592-c296-42f8-b47f-8ec9ae5ebd6c Groups:[system:serviceaccounts system:serviceaccounts:kyverno system:authenticated] Extra:map[authentication.kubernetes.io/pod-name:[kyverno-reports-controller-d8b7d7498-qp8kn] authentication.kubernetes.io/pod-uid:[a7126f78-8eed-4d0e-9e6e-9902cd6728e0]]} Audiences:[https://kubernetes.default.svc] Error:}}
2023-12-07T15:31:54.204Z	INFO	verifier/client.go:188	Request recieved with data={ImageReferences:[123456662101.dkr.ecr.eu-west-3.amazonaws.com*] Images:{InitContainers:map[] Containers:map[boweb-iam-frontend:{ImageInfo:{Registry:123456662101.dkr.ecr.eu-west-3.amazonaws.com Name:myaws1m-iam-frontend Path:myaws1m-iam-frontend Tag:0.1.6 Digest:} Pointer:/spec/template/spec/containers/0/image}] EphemeralContainers:map[]} TrustPolicy:mynet-trust-policy Attestations:[] Metadata:}
2023-12-07T15:31:54.204Z	INFO	notationfactory/client.go:91	Using trust policy provided in the request mynet-trust-policy
2023-12-07T15:31:54.204Z	INFO	notationfactory/client.go:102	Found notation verifer for trust policy mynet-trust-policy
2023-12-07T15:31:54.204Z	INFO	cache/client.go:120	Getting image from the cache: trustPolicy=mynet-trust-policy, imageRef=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:31:54.204Z	INFO	cache/client.go:139	Entry found in the cache mynet-trust-policy;123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6 entry={{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 sha256:88d3093e15aa236813d14d346c8a0c0349459e35cc894dc86a2ad5a07cff3e32} /spec/containers/0/image}
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:359	Entry for the image found in cache, skipping image={{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}; trustpolicy=mynet-trust-policy
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:125	verified map[%!d(string=myaws1m-iam-frontend):{{%!d(string=123456662101.dkr.ecr.eu-west-3.amazonaws.com) %!d(string=myaws1m-iam-frontend) %!d(string=myaws1m-iam-frontend) %!d(string=0.1.6) %!d(string=)} %!d(string=/spec/template/spec/containers/0/image)}] containers 
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:145	verified map[] initContainers
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:165	verified map[] ephemeralContainers
2023-12-07T15:31:54.204Z	INFO	verifier/response.go:109	building attestation set []
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:171	built attestation list%!(EXTRA map[string]types.AttestationList=map[123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6:map[]])
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:181	verifying attestations map[123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6:map[]]
2023-12-07T15:31:54.204Z	INFO	verifier/verify.go:191	verifying attestation, image=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6; attestations=map[]
2023-12-07T15:31:54.204Z	INFO	verifier/response.go:104	Sending response result=[{Operation:replace Path:/spec/containers/0/image Value:123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend@sha256:88d3093e15aa236813d14d346c8a0c0349459e35cc894dc86a2ad5a07cff3e32}]
2023-12-07T15:31:54.204Z	INFO	verifier/client.go:226	Sending response {

Using Pod Identity association :

2023-12-07T15:49:43.093Z	INFO	verifier/client.go:175	Token is authorized {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name: GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[{Manager:kyverno-notation-aws Operation:Update APIVersion:authentication.k8s.io/v1 Time:2023-12-07 15:49:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:spec":{"f:token":{}}} Subresource:}]} Spec:{Token:eyJhbGciOiJSUzI1NiIsImtpZCI6IjQ5ODk4ZmUzYzQ1YmIyMWRhMjM5MzkyZGIxOTI0ZmUzYzVjNGQ2OWQifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTczMzQ5OTQzMywiaWF0IjoxNzAxOTYzNDMzLCJpc3MiOiJodHRwczovL29pZGMuZWtzLmV1LXdlc3QtMy5hbWF6b25hd3MuY29tL2lkLzcwN0Q4NDc0OTREMUQ0ODA4QjI4RDYzQTJCRDBEQUY3Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJreXZlcm5vIiwicG9kIjp7Im5hbWUiOiJreXZlcm5vLXJlcG9ydHMtY29udHJvbGxlci1kOGI3ZDc0OTgtcXA4a24iLCJ1aWQiOiJhNzEyNmY3OC04ZWVkLTRkMGUtOWU2ZS05OTAyY2Q2NzI4ZTAifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6Imt5dmVybm8tcmVwb3J0cy1jb250cm9sbGVyIiwidWlkIjoiNTg4ODY1OTItYzI5Ni00MmY4LWI0N2YtOGVjOWFlNWViZDZjIn0sIndhcm5hZnRlciI6MTcwMTk2NzA0MH0sIm5iZiI6MTcwMTk2MzQzMywic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt5dmVybm86a3l2ZXJuby1yZXBvcnRzLWNvbnRyb2xsZXIifQ.KX6KzUAfW4hD6mQRKqlCg6yU16hvtW71w-C8TMqZRh7ti--Gn5EDyI6O3Za-4dqANS8t6ezB1BHd7TwIXEcBsxibFWoBc72eb5cjAOK5RcC2sy-7jCi0jpxXQ3D8dJD5dibVCiwaeCUs017SVnd9rIcYrfng6ClBx8Jp02s2cOdx619MGvQM-taoF7Xtp3e-wG7hzPly8kqWXY4UZWqRBvRUZCvNR4ISjGm9XlWDqeYPkMLzSCjr-LjOQN3Ou5ZnrZeEcmKMDYbgVAz3Y3v4RsZKGj2CAV2lGyzr3-SirHP6wqNjSbg Audiences:[]} Status:{Authenticated:true User:{Username:system:serviceaccount:kyverno:kyverno-reports-controller UID:58886592-c296-42f8-b47f-8ec9ae5ebd6c Groups:[system:serviceaccounts system:serviceaccounts:kyverno system:authenticated] Extra:map[authentication.kubernetes.io/pod-name:[kyverno-reports-controller-d8b7d7498-qp8kn] authentication.kubernetes.io/pod-uid:[a7126f78-8eed-4d0e-9e6e-9902cd6728e0]]} Audiences:[https://kubernetes.default.svc] Error:}}
2023-12-07T15:49:43.093Z	INFO	verifier/client.go:188	Request recieved with data={ImageReferences:[123456662101.dkr.ecr.eu-west-3.amazonaws.com*] Images:{InitContainers:map[] Containers:map[boweb-iam-frontend:{ImageInfo:{Registry:123456662101.dkr.ecr.eu-west-3.amazonaws.com Name:myaws1m-iam-frontend Path:myaws1m-iam-frontend Tag:0.1.6 Digest:} Pointer:/spec/template/spec/containers/0/image}] EphemeralContainers:map[]} TrustPolicy:mynet-trust-policy Attestations:[] Metadata:}
2023-12-07T15:49:43.093Z	INFO	notationfactory/client.go:91	Using trust policy provided in the request mynet-trust-policy
2023-12-07T15:49:43.093Z	INFO	notationfactory/client.go:102	Found notation verifer for trust policy mynet-trust-policy
2023-12-07T15:49:43.093Z	INFO	cache/client.go:120	Getting image from the cache: trustPolicy=mynet-trust-policy, imageRef=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z	ERROR	cache/client.go:130	Entry not found key=mynet-trust-policy;123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z	INFO	verifier/verify.go:362	Entry not found in the cache verifying image=123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z	INFO	verifier/verify.go:364	verifying image infos {ImageInfo:{Registry:123456662101.dkr.ecr.eu-west-3.amazonaws.com Name:myaws1m-iam-frontend Path:myaws1m-iam-frontend Tag:0.1.6 Digest:} Pointer:/spec/template/spec/containers/0/image}
2023-12-07T15:49:43.093Z	INFO	verifier/verify.go:382	verifying image 123456662101.dkr.ecr.eu-west-3.amazonaws.com/myaws1m-iam-frontend:0.1.6
2023-12-07T15:49:43.093Z	ERROR	verifier/verify.go:367	verification failed for image {{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}: failed to resolve digest: failed to retrieve credentials: failed to load default configuration: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
2023-12-07T15:49:43.093Z	ERROR	verifier/verify.go:116	failed to verify container myaws1m-iam-frontend: failed to verify image {{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}: failed to resolve digest: failed to retrieve credentials: failed to load default configuration: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed
2023-12-07T15:49:43.093Z	ERROR	verifier/response.go:79	Verification failed with error failed to verify container myaws1m-iam-frontend: failed to verify image {{123456662101.dkr.ecr.eu-west-3.amazonaws.com myaws1m-iam-frontend myaws1m-iam-frontend 0.1.6 } /spec/template/spec/containers/0/image}: failed to resolve digest: failed to retrieve credentials: failed to load default configuration: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed

I can see the credentials varaibles set on the Pod :

apiVersion: v1
kind: Pod
metadata:
  name: kyverno-notation-aws-78d96bcc75-6qhxk
  generateName: kyverno-notation-aws-78d96bcc75-
  namespace: kyverno-notation-aws
  uid: 37cbb3ce-ed4d-4c0d-992e-6ffe1fda7502
  resourceVersion: '66079498'
  creationTimestamp: '2023-12-07T15:47:58Z'
  labels:
    app: kyverno-notation-aws
    pod-template-hash: 78d96bcc75
  selfLink: >-
    /api/v1/namespaces/kyverno-notation-aws/pods/kyverno-notation-aws-78d96bcc75-6qhxk
spec:
  volumes:
    - name: aws-iam-token
      projected:
        sources:
          - serviceAccountToken:
              audience: pods.eks.amazonaws.com
              expirationSeconds: 86400
              path: token
        defaultMode: 420
    - name: notation
      emptyDir: {}
    - name: kube-api-access-dxfz2
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: kyverno-notation-aws
      image: >-
        ghcr.io/nirmata/kyverno-notation-aws:latest
      args:
        - '--debug'
        - '--cacheEnabled'
        - '--cacheMaxSize=2000'
        - '--cacheTTLDurationSeconds=7200'
      env:
        - name: NOTATION_DIR
          value: /notation
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: SERVICE_NAME
          value: svc
        - name: DEPLOYMENT_NAME
          value: kyverno-notation-aws
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AWS_REGION
          value: eu-west-3
        - name: DEFAULT_TRUST_POLICY
          value: aws-signer-trust-policy
        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
          value: http://169.254.170.23/v1/credentials
        - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      resources:
        limits:
          memory: 512Mi
        requests:
          cpu: 100m
          memory: 32Mi
      volumeMounts:
        - name: notation
          mountPath: /notation
        - name: kube-api-access-dxfz2
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        - name: aws-iam-token
          readOnly: true
          mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: Always
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsUser: 2000
        runAsGroup: 3000
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
  restartPolicy: Always
  terminationGracePeriodSeconds: 5
  dnsPolicy: ClusterFirst
  serviceAccountName: kyverno-notation-aws
  serviceAccount: kyverno-notation-aws
  securityContext:
    runAsNonRoot: true
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority
@vishal-chdhry vishal-chdhry self-assigned this Jan 19, 2024
@calvinbui
Copy link

it should work now that aws-sdk v2 has been upgraded.

@vishal-chdhry is the latest untagged tag safe to use? we are still using the commit that corresponds to the v1 tag.

@vishal-chdhry
Copy link
Contributor

@calvinbui yes you can use latest, it is safe to use

@calvinbui
Copy link

calvinbui commented Oct 22, 2024 •
edited
Loading

thanks @vishal-chdhry the latest tag latest@sha256:c8ee5afd88cb1d6c4f0d27c9fb5581982841ca1ad9be742a1095cdcb89de60cc works - but eks pod identity is still not working. same error as OP listed.

failed to execute the verify-signature command for plugin com.amazonaws.signer.notation.plugin: ERROR: invalid endpoint host, "169.254.170.23", only loopback hosts are allowed

i believe the problem is the aws-signer plugin's dependency is out of date based on the error message. the version of their website is 1.0.298, GitHub is 1.0.350. i'll test this out.

@vishal-chdhry
Copy link
Contributor

@calvinbui I think you are right
The minimum SDK version for golang is a release from November 2023: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html

The signer binary was last updated in June 2023: Bin Download Page
Changelog: https://d2hvyiie56hcat.cloudfront.net/CHANGELOG

@calvinbui calvinbui mentioned this issue Nov 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vishal-chdhry vishal-chdhry

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@calvinbui @vishal-chdhry @Bart-Leboeuf