Automatically create S3 backend bucket from pipeline

[gberlot-luna]

I'm running a new plan on a dev branch that will create a new VPC in another region using the github action pipeline. I notice that isn't able to create the corresponding S3 bucket and the plan failed. Should I have to run terragrunt init manually on that resource in order to get the bucket created ? See log below:
.
.
[ecs-deploy-runner][2023-06-13T21:42:20+0000] [INFO] [infrastructure-deploy-script] 2023-06-13 21:42:20 Running command "terragrunt plan -input=false"
[ecs-deploy-runner][2023-06-13T21:42:23+0000] Remote state S3 bucket lunasolutions-dev-us-east-2-tf-state does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) time=2023-06-13T21:42:23Z level=error msg=EOF
[ecs-deploy-runner][2023-06-13T21:42:23+0000] time=2023-06-13T21:42:23Z level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1
[ecs-deploy-runner][2023-06-13T21:42:23+0000] [INFO] [infrastructure-deploy-script] 2023-06-13 21:42:23 "terragrunt plan" exited with code 1
[ecs-deploy-runner][2023-06-13T21:42:23+0000] Error: Could not run "terragrunt plan". Error=Remote state S3 bucket lunasolutions-dev-us-east-2-tf-state does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) time=2023-06-13T21:42:23Z level=error msg=EOF
[ecs-deploy-runner][2023-06-13T21:42:23+0000] time=2023-06-13T21:42:23Z level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1
[ecs-deploy-runner][2023-06-13T21:42:23+0000] exit status 1
.
.
Please, advice.

---

Tracked in ticket #110254

[gitsstewart]

@gberlot-luna can you check to make sure you have all the latest changes and versions outlined in this post regarding private S3 buckets and ACLs : https://github.com/orgs/gruntwork-io/discussions/701

[arsci]

Hi @gberlot-luna, could you provide some additional information:

• Can you describe the configuration changes you made in your branch to add the new VPC in the new region?

• From the support ticket you submitted, it sounds like after you received the above terragrunt error you tried running it locally, and the new state bucket was created in the new region, but you received the S3 AccessControlListNotSupported error. It sounds like at this point the new S3 bucket got created, but was not fully configured at the point the error popped up. Is this correct?

• You had mentioned you also changed the Bucket Policy and Bucket Ownership - was this done through the AWS console?

• Did any other resources (the VPC) get created, or is it just the S3 bucket so far?

[gberlot-luna]

Hi:

1. At the same level of us-east-1, I copied that folder and renamed to us-east-2, delete everything and only leave region.hcl (I changed the region within this file for us-east-2) and keep folder dev/networking/vpc and the terragrunt.hcl file.
2. when running terragrunt init the following error was displayed, but the tf-state and tf-logs buckets were created after pressing yes:

ERRO[0024] Create S3 bucket with retry lunasolutions-dev-us-east-2-tf-state returned an error: AccessControlListNotSupported: The bucket does not allow ACLs

        status code: 400, request id: N89RY57PKRM4GNE7, host id: wP1AWOYgbG299xKYCARMJiVg6s6j9Zf77xjMahuvc0TEJrDuc5eGYvKREVgwmb0fz2Svd1Vc2rA=. Sleeping for 10s and will try again.

I noticed that the new tf-state bucket don't have the same configuration as the one in us-east-1, So, Using the aws console, I changed the buckets policy(copy from a existing one in us-east-1 to us-east-2 bucket) and changed Object Ownership, from Bucket owner preferred to Object writer(enabled ACL).

I was able to run:

• terragrunt plan and apply from the command line on my laptop. (No errors displayed and 162 resources were created)

but, when trying to run using the ECS deploy runner, there's no way of the buckets to get created, and I if a run terraform init in order to get buckets created and then when trying to run plan using ECS deploy runner the following error was displayed all the time in ECS:

Remote state S3 bucket lunasolutions-dev-us-east-2-tf-state is out of date. Would you like Terragrunt to update it? (y/n) time=2023-06-15T18:25:44Z level=error msg=EOF

[arsci]

Hi @gberlot-luna,

It looks like there are two main components here:

1. The first issue is the ACL error. This is originating from changes AWS made a few months ago to default S3 configurations. We've fixed this is most of the modules and tools, so I suspect you may be running into this error due to an outdated version somewhere.

   Can you check which version of terragrunt you are using on your local machine, and then check which version of terragrunt is being executed in the ECS deploy runner?

   I have a feeling the version of terragrunt you are using locally may be a newer version with the S3 ACL updates (which is likely the reason you did not encounter this error locally), and the ECS deploy running is probably on an older version.

2. I believe the second error you are seeing with the state bucket is a result of executing terragrunt and creating resources from your local machine, then also making manual changes in the AWS console. This means the state file is out of sync with what actually exists in AWS. When terragrunt runs it is recognizing this mismatch and giving the error that it is out of date and is asking to update it. When you run terragrunt locally do you see the state error, or is that error only present with the ecs deploy runner?

[gberlot-luna]

Hi @arsci

1- Terragrunt Version:
Local Machine: v0.36.0
ECS Deploy Runner: Can you explain how to get it ? In case we need to update can you provide a link that explains how to do it ?

2- I don't agree with the second statement, for 2 reasons:
a) Error is only present with the ecs deploy runner.
b) I'm getting the error even when the S3 state bucket is empty, here's the steps to reproduce:

• run a terragrunt int (bucket TF-STATE is created with the error I mentioned before)
• Apply manually changes from console in buckets (Policy and Object Ownership)
• Until here the bucket is empty, no resources were created
• Run ECS deploy runner to create the new VPC.
• The following error is displayed: Remote state S3 bucket lunasolutions-dev-us-east-2-tf-state is out of date. Would you like Terragrunt to update it? (y/n) time=2023-06-15T18:25:44Z level=error msg=EOF

[arsci]

Ah, I think I understand what's happening @gberlot-luna.

It looks like your ecs-deploy-runner is on v0.45.11, this can be checked by looking at the build_deploy_runner_image.sh build arguments. Looking at your infrastructure-live repo I can see this being passed in: --build-arg 'terragrunt_version=v0.45.11' \ along with terraform version 1.1.9 from --build-arg 'terraform_version=1.1.9' \

What is likely happening is you deployed locally using v0.36.0, which is over a year old. This also explains the ACL error you were seeing (as this was fixed in terragrunt v0.45.4).

Terragrunt state currently is expecting v0.36.0 since that's what it was created with, but the ecs-deploy-runner is on a much newer version. If you update your local version of terragrunt to v0.45.11 to match what the ecs-deploy-runner is using, along with your local version of terraform to 1.1.9 I suspect you will see the out of date error with the state bucket, and you'll be prompted to update it. Once you update it locally and let terragrunt update the state, you can try re-running the ecs-deploy-runner and it should hopefully work.

[gberlot-luna]

@arsci I just changed the version of terrargrunt (locally) to the same one of ECS deploy runner and I didn't face any error on executing the terraform plan from ECS deploy runner. Thanks!!!

Last question prior to close, every time we add a new region that requires a new bucket creation as part of the backend, we need to execute a terragrunt int locally since this functionality isn't supported by ECS deploy runner, right ?

[arsci]

Hey @gberlot-luna, great to hear that fixed it! Yes it looks like you would need execute locally once to first setup the state bucket