Repeated 'TooManyRequestsException' during EKS creation #193
Comments
|
I receive the same error. Tried to use Services: creating CloudFormation changeset...
[██████████████████████████████████████████████████████▋···] (179/190)
5:18:22 PM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | petsite/AwsAuth/ma...t/Resource/Default
Received response status [FAILED] from custom resource. Message returned: TooManyRequestsException: Rate Exceeded.
at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: 80fbe0a9-1d21-415d-bdf5-7e3a55a7782e)
5:18:22 PM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | ssmAgentdeployment/Resource/Default
Received response status [FAILED] from custom resource. Message returned: TooManyRequestsException: Rate Exceeded.
at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: b64117d3-793e-47cc-84f0-f3f43740de95)
5:18:24 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | Services
The following resource(s) failed to create: [LBServiceAccount67A900BC, loadBalancerServiceAccount1D19AD3A, ssmAgentdeployment3666D473, xraydeployment7059A3EB, petsiteAwsAuthmanifest576F9E03, k8sdashboardrbac66807D7E, loadB
alancerCRDEF35F65C, cloudwatcheployment7FE6A515]. Rollback requested by user.
5:18:24 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | Services
The following resource(s) failed to create: [LBServiceAccount67A900BC, loadBalancerServiceAccount1D19AD3A, ssmAgentdeployment3666D473, xraydeployment7059A3EB, petsiteAwsAuthmanifest576F9E03, k8sdashboardrbac66807D7E, loadB
alancerCRDEF35F65C, cloudwatcheployment7FE6A515]. Rollback requested by user.
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--queueurl
5:18:33 PM | DELETE_IN_PROGRESS | AWS::ECS::Service | list-adoptions-ser...ce/Service/Service
5:18:33 PM | DELETE_IN_PROGRESS | AWS::IAM::Policy | pay-for-adoption-s...Role/DefaultPolicy
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--petadoptionsstepfnarn
5:18:33 PM | DELETE_IN_PROGRESS | AWS::EC2::SecurityGroupIngress | ECSServicesSG/from...tyGroup4D28D01D:80
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --eks--petsite--OIDCProviderArn
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--updateadoptionstatusurl
5:18:33 PM | DELETE_IN_PROGRESS | AWS::IAM::Policy | search-service/taskRole/DefaultPolicy
5:18:33 PM | DELETE_IN_PROGRESS | Custom::AWSCDK-EKS-KubernetesResource | xraydeployment/Resource/Default
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--dynamodbtablename
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--searchapiurl
5:18:33 PM | DELETE_IN_PROGRESS | AWS::IAM::Policy | traffic-generator-...Role/DefaultPolicy
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--cleanupadoptionsurl
5:18:33 PM | DELETE_IN_PROGRESS | AWS::IAM::Policy | list-adoptions-ser...Role/DefaultPolicy
5:18:33 PM | DELETE_IN_PROGRESS | AWS::CDK::Metadata | @aws-cdk--aws-eks....DKMetadata/Default
5:18:33 PM | DELETE_IN_PROGRESS | AWS::Lambda::Permission | status-updater-ser...ater7EE91D69.ANY..
5:18:33 PM | DELETE_IN_PROGRESS | AWS::EC2::SecurityGroupIngress | ECSServicesSG/from...tyGroupCBA33A31:80
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --eks--petsite--OIDCProviderUrl
5:18:33 PM | DELETE_IN_PROGRESS | Custom::AWSCDK-EKS-KubernetesResource | petsite/AwsAuth/ma...t/Resource/Default
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--payforadoptionmetricsurl
5:18:33 PM | DELETE_IN_PROGRESS | AWS::ECS::Service | traffic-generator-...ce/Service/Service
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--stackname
5:18:33 PM | DELETE_IN_PROGRESS | AWS::SSM::Parameter | --petstore--snsarn
5:18:33 PM | DELETE_IN_PROGRESS | Custom::AWSCDK-EKS-KubernetesResource | ssmAgentdeployment/Resource/Default
^CIf the nested cloud formation stacks were to be deployed with time separation, or manual confirmation, perhaps it could help. Rollback takes time, but succeeds. +1 on being atomic. |
|
I had the same issue; this worked for me:
|
|
@edmundcraske-bjss , I couldn't personally test this solution by @dastra above because I don't get this error at all. Would you be interested in checking this? |
I'm afraid that I don't really have a lot of time to test this, sorry. |
|
The only change to the command i made was to include --no-rollback parameter, as i wanted to continue where it left off when i got the rate limit exceeded error rather than rolling back and trying again. However when i added --no-rollback the command completed successfully. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to set up the workshop in my own AWS account using the instructions at https://catalog.us-east-1.prod.workshops.aws/workshops/31676d37-bbe9-4992-9cd1-ceae13c5116c/en-US/installation/not-using-ee/deploy-app but every time I try to do the deployment at step 16, when running
cdk deploy --context admin_role=$EKS_ADMIN_ARN Services --context dashboard_role_arn=$CONSOLE_ROLE_ARN --require-approval neverit fails with a Rate Exceeded error and rolls back - taking ages both to get to the failure and to roll back. I've no idea what I could do to fix it.The text was updated successfully, but these errors were encountered: