❗Notice: CDK CLI: "The security token included in the request is invalid" when using user credentials since 2.167.0

[AdamPD]

Please add your +1 👍 to let us know you have encountered this

Status: RESOLVED

Overview:

In version 2.167.0 CLI commands, including deployments, fail with authentication errors.

This main cause is with authentication configurations that do not use AWS_SESSION_TOKEN, like IAM User credentials.
A second cause is related to the location of the region configuration, see #32130

Complete Error Message:

The security token included in the request is invalid

Workaround:

Revert to 2.166.0

Solution:

Upgrade to 2.167.1

Related Issues:

#32130

---

Original issue

Since version 2.167.0, deployments fail due to the inability to get the AWS account ID. The following error occurs in the verbose output of cdk synthesize/deploy:

[01:50:17] Resolving default credentials
[01:50:17] Looking up default account ID from STS
[01:50:18] Unable to determine the default AWS account (InvalidClientTokenId): The security token included in the request is invalid

This is despite having correct AWS environment variables set (AWS_DEFAULT_REGION, AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID) and aws sts get-caller-identity works correctly.

Rolling back to 2.166.0 with:

npm install -g [email protected]

resolves the issue and deployments resume as per normal.

Regression Issue

Confirmed Regression.

Last Known Working CDK Version

No response

Expected Behavior

CDK should function correctly and retrieve the account ID via the AWS credentials.

Current Behavior

CDK throws an error stating that it cannot retrieve the account ID due to security token issues.

Reproduction Steps

Upgrade to 2.167.0, use AWS environment variable credentials but don't specify the account ID, and run cdk synthesize.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.167.0 (build 677e108)

Framework Version

No response

Node.js Version

v22.6.0

OS

Ubuntu 22.04.4 LTS

Language

.NET

Language Version

No response

Other information

No response

[Neuroforge]

I got the same issue in Github Actions....

          name: Bootstrap Backend with CDK (Production)
                env:
                  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID_PROD}}
                  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY_PROD}}
                  CDK_DEPLOY_ACCOUNT: ${{ secrets.CDK_DEPLOY_ACCOUNT_PROD}}
                  CDK_DEPLOY_REGION: ${{ secrets.AWS_REGION_PROD}}
                  AWS_REGION: ${{ secrets.AWS_REGION_PROD}}
                working-directory: backend
                run: npx cdk bootstrap aws://$CDK_DEPLOY_ACCOUNT/$CDK_DEPLOY_REGION`

Errors with...

❌ Environment aws:/// failed bootstrapping: Error: Need to perform AWS calls for account ***, but no credentials have been configured

@AdamPD's solution worked for me but isn't suitable longterm.

Is this related?

#21562

[Eonfuzz]

It's affecting our Azure cdk deploy pipelines too.

[RWS-RHC]

Also* affecting us on Bitbucket Pipelines.

[christiandunn]

I have noticed this issue on our CI/CD pipelines as well. AWS credential configuration that has worked before has stopped working. Confirmed it was on version 2.167.0.

[IgorPietraszko]

Same here. Downgraded to 2.166.0 and trying out.

[minimum-hsu]

In CDK 2.167.0, got an error meesage

botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the DescribeRepositories operation: The security token included in the request is invalid

After downgrading, the cdk diff and deploy jobs are functioning correctly.

[christiandunn]

Downgrading to the previous version also fixed the problem for me.

[raducostea]

Same for me, i have aws environment set up correctly. Downgrading to 2.166.0 fixed the issue.

Need to perform AWS calls for account ***, but no credentials have been configured

[otaviomacedo]

@AdamPD (or anyone else in this thread) can you please paste the output of cdk deploy -vvv?

[scarytom]

@otaviomacedo here is the -vvv output from a cdk diff, which works fine on 2.166.0

[09:08:41] [trace] SdkProvider#resolveEnvironment()
[09:08:41] [trace] SdkProvider#baseCredentialsPartition()
[09:08:41] [trace]   SdkProvider#resolveEnvironment()
[09:08:41] [trace]   SdkProvider#obtainBaseCredentials()
[09:08:41] [trace]     SdkProvider#defaultAccount()
[09:08:41] [trace]     SdkProvider#defaultCredentials()
[09:08:41] [trace]   SDK#currentAccount()
[09:08:41] Retrieved account ID YYYYYYYYYYYY from disk cache
[09:08:41] [trace] SdkProvider#forEnvironment()
[09:08:41] [trace]   SdkProvider#resolveEnvironment()
[09:08:41] [trace]   SdkProvider#obtainBaseCredentials()
[09:08:41] [trace]     SdkProvider#defaultAccount()
[09:08:41] [trace]     SdkProvider#defaultCredentials()
[09:08:41] [trace]   SdkProvider#withAssumedRole()
[09:08:41] Assuming role 'arn:aws:iam::XXXXXXXXXXXX:role/cdk-sbsl-infra-lookup-role-XXXXXXXXXXXX-us-east-1'.
[09:08:42] Assuming role failed: The security token included in the request is invalid
Could not assume role in target account using current credentials (which are for account YYYYYYYYYYYY) The security token included in the request is invalid . Please make sure that this role exists in the account. If it doesn't exist, (re)-bootstrap the environment with the right '--trust', using the latest version of the CDK CLI.
[09:08:42] [trace] SdkProvider#resolveEnvironment()
[09:08:42] [trace] SdkProvider#baseCredentialsPartition()
[09:08:42] [trace]   SdkProvider#resolveEnvironment()
[09:08:42] [trace]   SdkProvider#obtainBaseCredentials()
[09:08:42] [trace]     SdkProvider#defaultAccount()
[09:08:42] [trace]     SdkProvider#defaultCredentials()
[09:08:42] [trace]   SDK#currentAccount()
[09:08:42] Retrieved account ID YYYYYYYYYYYY from disk cache
[09:08:42] [trace] SdkProvider#forEnvironment()
[09:08:42] [trace]   SdkProvider#resolveEnvironment()
[09:08:42] [trace]   SdkProvider#obtainBaseCredentials()
[09:08:42] [trace]     SdkProvider#defaultAccount()
[09:08:42] [trace]     SdkProvider#defaultCredentials()
[09:08:42] [trace]   SdkProvider#withAssumedRole()
[09:08:42] Assuming role 'arn:aws:iam::XXXXXXXXXXXX:role/cdk-sbsl-infra-deploy-role-XXXXXXXXXXXX-us-east-1'.
[09:08:43] Assuming role failed: The security token included in the request is invalid
[09:08:43] Reading cached notices from /home/tom/.cdk/cache/notices.json
Could not assume role in target account using current credentials (which are for account YYYYYYYYYYYY) The security token included in the request is invalid . Please make sure that this role exists in the account. If it doesn't exist, (re)-bootstrap the environment with the right '--trust', using the latest version of the CDK CLI.
[09:08:43] Error: Could not assume role in target account using current credentials (which are for account YYYYYYYYYYYY) The security token included in the request is invalid . Please make sure that this role exists in the account. If it doesn't exist, (re)-bootstrap the environment with the right '--trust', using the latest version of the CDK CLI.
    at SdkProvider.withAssumedRole (/usr/lib/node_modules/aws-cdk/lib/index.js:593:5643098)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async SdkProvider.forEnvironment (/usr/lib/node_modules/aws-cdk/lib/index.js:593:5639480)
    at async EnvironmentAccess.cachedSdkForEnvironment (/usr/lib/node_modules/aws-cdk/lib/index.js:647:7329)
    at async EnvironmentAccess.prepareSdk (/usr/lib/node_modules/aws-cdk/lib/index.js:647:6285)
    at async Deployments.readCurrentTemplateWithNestedStacks (/usr/lib/node_modules/aws-cdk/lib/index.js:647:8697)
    at async CdkToolkit.diff (/usr/lib/node_modules/aws-cdk/lib/index.js:647:201596)
    at async exec3 (/usr/lib/node_modules/aws-cdk/lib/index.js:650:18965)

[scarytom]

Presume this is caused by #31702, but just a hunch

[otaviomacedo]

@scarytom Yes, it was. In your case, the CLI is getting credentials for one account, but trying to call another. Is there anything you can share, like your ~/.aws/config file (with confidential data redacted, of course)?

[otbe]

For us the issue is in the region, e.g.

deploy -vvv

10:38:42] Looking up AWS region in the EC2 Instance Metadata Service (IMDS).
[10:38:43] Unable to retrieve AWS region from IMDS: Error: Error fetching metadata token: TimeoutError: Connection timed out after 1000 ms
[10:38:43] Unable to determine AWS region from environment or AWS configuration (profile: "default"), defaulting to 'us-east-1'

Not quite sure how it worked in older aws-cdk versions but the region is specified in .aws/credentials similar to this

[default]
aws_access_key_id=
aws_secret_access_key=
aws_session_token=
aws_security_token=
region=eu-central-1

That means our stacks are deployed in the wrong region with 2.167.0 🫠
Reverting back to 2.166.0 solves the issue.

[rix0rrr]

@otbe:

1. Do you also have a ~/.aws/config file?
2. Were you running this code on an EC2 instance?

[AdamPD]

To be clear on my side, I am running this within AppVeyor on a clean build each time, and we do not have any ~/.aws directory with any configuration - our configuration is 100% through environment variables.

[rix0rrr]

Can confirm that if you do not have a ~/.aws directory on your machine and configure using AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, it fails on my machine as well.

Nevermind I typo'ed the region ☹️. Actually works fine.

[scarytom]

@otaviomacedo no, we don't us an ~/.aws/config file. Our CI system just sets the AWS credentials via AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars, and our CDK Environment then picks one of our other AWS accounts that has been cdk-bootstrapped to allow access via RBAC.

This setup has been working fine for years, and now fails with 2.167.0

[otbe]

@rix0rrr

 1. Do you also have a `~/.aws/config` file?

2. Were you running this code on an EC2 instance?

1. no ~/.aws/config
2. Yes it was executed on EC2 (however in a docker container)

[rix0rrr]

There are 2 distinct problems being discussed in this thread. While they are both introduced by switching from SDKv2 to SDKv3 we should make sure to distinguish them.

• The security token included in the request is invalid: seems to happen in some circumstances on CI machines, where configuration happens entirely through environment variables.
• Unable to determine AWS region from environment or AWS configuration (profile: "default"), defaulting to 'us-east-1': the region that is present in a credentials file is not picked up.

This issue was originally created for the first issue by @AdamPD. @otbe, I will make a new issue for your error so we can discuss that there.

Edit: new issue here: #32130

[rix0rrr]

On the The security token included in the request is invalid error: it occurs to me that the users reporting this seem NOT to be using session credentials, but IAM User credentials instead (access key pair without a session token).

Maybe that's a lead...

[rix0rrr]

Woohoo, this reproduces! IAM User credentials seem broken:

$ ls ~/.aws
ls: /Users/huijbers/.aws: No such file or directory

$ env | grep AWS_
AWS_ACCESS_KEY_ID=*****
AWS_DEFAULT_REGION=us-east-2
AWS_SECRET_ACCESS_KEY=*******

$ npx cdk deploy -vvv
...
[11:33:04] Unable to determine the default AWS account (InvalidClientTokenId): The security token included in the request is invalid
...
Unable to resolve AWS account to use. It must be either configured when you define your CDK Stack, or through the environment

[mrgrain]

We have deprecated [email protected] on npmjs.com while we continue our investigation.

Please note The aws-cdk-lib and other @aws-cdk/* packages are not effected by the deprecation.

[gsingh1]

I can confirm the same issue happening on GHE instances too. NPM package deprecation is working as expected and our builds are clearing on 2.166.0. Thanks.

[ashishdhingra]

Another related issues:

• CDK CLI: region must be in config file (ignored in credentials file) since 2.167.0 #32130
• (credentials): (Need to perform AWS calls for account ..., but no credentials have been configured ) #32135

[ashishdhingra]

Another report:

• (credentials): assume role credentials in aws/config do not work since 2.167.0 #32137

[mrgrain]

Hi all, we have released v2.167.1 with fixes for the reported auth issues.

Please try again and report back. Reports that everything is working are equally useful as failures.