Support for EKS Pod Identity

[askulkarni2]

Describe the question/issue

Amazon EKS launched Pod Identity to simplify IAM configuration for pods. In order for AWS for FluentBit to support this, it will need to use an AWS SDK version that provides the new credential provider logic available >= v1.47.11.

[arvind-v]

To build on @askulkarni2's comment above: Both EKS pod identity and IAM role for Service Account (IRSA) are not currently working with AWS for FluentBit. I tested an OpenSearch Ingestion Pipeline with AWS for FluentBit following the instructions at https://docs.aws.amazon.com/opensearch-service/latest/developerguide/configure-client-fluentbit.html.

For http output, FluentBit seems to build the SIGv4 request from scratch. IRSA and Pod Identity are ignored and the code relies on aws_role_arn to build the request. Further, the node group instance role is used for signing the SIGv4 request instead of the IAM role associated with the EKS pod identity or IRSA. See debug output from fluenbit pod below:

[2024/01/30 23:40:33] [debug] [aws_client] Unable to parse API response- response is not valid JSON.
[2024/01/30 23:40:33] [debug] [aws_credentials] STS raw response: 
<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>AccessDenied</Code>
    <Message>User: arn:aws:sts::nnnnnnnn:assumed-role/eksctl-eks-cluster-name-nodegroup-NodeInstanceRole-abcdef/i-nnnnn is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::nnnnnnnn:role/ingestion-role</Message>
  </Error>
  <RequestId>afd30be4-0342-47fc-ba2d-78144185b411</RequestId>
</ErrorResponse>

[2024/01/30 23:40:33] [error] [aws_credentials] STS assume role request failed

A temporary workaround was to setup the aws_role_arn, which is named ingestion-role in my example, with a trust policy that enables the EKS cluster's NodeInstanceRole to assume the role.

A change in upstream flb_aws_credentials_sts.c may be needed to build the SIGv4 request differently and support EKS pod identity and IRSA.

[PettitWesley]

I created the AWS credential providers for the Fluent Bit C plugins and docs: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#credential-chain-resolution-issues

These are our different plugins: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins

I'm new to EKS Pod identities. It looks like the credentials are vended via a local agent that has an http cred endpoint on a link local IP: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

This is exacty the same as how ECS cred vending works AFAICT: https://aws.amazon.com/blogs/compute/under-the-hood-task-networking-for-amazon-ecs/

So it should be pretty simple to modify the http credentials provider to support it: https://github.com/fluent/fluent-bit/blob/master/src/aws/flb_aws_credentials_http.c

I can't promise I'll work on this right now but I agree that Fluent Bit should support all new EKS features.

[PettitWesley]

Possibly related: aws/amazon-eks-pod-identity-webhook#216 (comment)

[jefchien]

Took a look at the AWS SDK Go implementation and it's similar to the ECS credential provider with a few differences. It is in front of the ECS credential provider in the chain and selected based on the presence of AWS_CONTAINER_CREDENTIALS_FULL_URI (which contains the URI to make the request to). EKS Pod Identities support requires authentication token providers, which are also configured via environment variable. The rest of the HTTP request and response parsing is the same.

[TreeKat71]

Any update? Seems like IRSAv2 is not working right now

[PettitWesley]

I am working on implementing support for EKS Pod identity actively. I can not promise a release date for it.

[georgejohnis]

Thank you @PettitWesley for working on it.

[iamahgoub]

@PettitWesley -- any update on this work? Thanks!

[trmbdevops]

I am working on implementing support for EKS Pod identity actively. I can not promise a release date for it.

@PettitWesley Any update on this yet?

[voidlily]

I'm also interested in this in order to get the container insights addon off of needing cloudwatch permissions on the instance profile and being able to use pod identity instead.

This may require an update to fluent-bit upstream, see these linked issues

fluent/fluent-bit#8550
fluent/fluent-bit#9206

[AzySir]

I am working on implementing support for EKS Pod identity actively. I can not promise a release date for it.

Thanks for that! Do you think it's worth putting a giant label on the READ ME - so newbies like me don't spend hours on it :)