Setting up infrastructure through a CI pipeline

[flixman]

Hi everybody! We are trying to get a CI pipeline to set up our infrastructure. Given a commit with additive changes (provisioning / updating infrastructure) that is easy to do: terragrunt run-all apply, and done. However, for destructive changes, this is more complicated: The folder with the component to be removed is just not there, so terragrunt does not know about it. We are evaluating now a course of action based on either:

1. analyse the last commit, looking for removed files and then verifying that their folder is indeed gone. Then run terragrunt destroy.
2. getting a list of the state files in the store backend, compare that to the current list of folders, download-init-destroy the ones that are in the former and not in the later.

Do you guys know if there is a better way to do this? We are looking at it from different angles, and cannot find one. Also, doing this feels too intricate.

[yhakbar]

Hey @flixman ,

I can tell you what Gruntwork Pipelines does.

When determine what actions to take, it knows that there are three git references that are relevant, the source, target and post-merge refs.

It can get really confusing to talk about which of these is what, so for the sake of this explanation, think of a pull request that is trying to remove one directory with a terragrunt.hcl file in it (we call this a unit) named foo, and add another named bar. The pull request is trying to merge branch my-changes into main. In this example, we can call my-changes the source ref and main before the merge the target ref. After the merge, there's one more ref that's relevant, which is the post-merge ref.

Whenever terragrunt.hcl files are added, changed or renamed, we inspect the git diff and decide to run terragrunt plan in the source ref prior to merge, and terragrunt apply in the post-merge ref.

Whenever terragrunt.hcl files are removed or renamed, we inspect the git diff and decide to run terragrunt plan -destroy in the target ref (main before the merge), and terragrunt destroy in the target ref (main before the merge).

The end result is that you destroy stuff in the git context of your code before your removal of files, and you add stuff in the git context of your code after you add/change files.

The details there about renaming have to do with something you might run into as you start to implement, which is that if you remove a directory named foo and add the same contents in another directory named bar, git will be smart and consider it a directory rename, not a destroy/add. You want to untangle that so that you can properly destroy the removed configurations and apply the added ones.

[flixman]

hey @yhakbar Thank you for your answer! your approach has some similarities with what we had in place, analysing the commits submitted since the last successful run of the pipeline. However, there is something I do not understand from your answer: why not just...
... When removing, plan and destroy from the source ref
... When modifying/updating, plan and apply from the post-merge ref
?

[yhakbar]

(Using my definitions above), the source ref no longer has the files that you would be running plan -destroy and destroy with, so you have to use the ref where they still exist.

The plan/apply, etc approach would work. You just need to create a temporary ephemeral commit in CI for the post-merge ref during the PR phase, as you won't have a merge commit at that stage. You'll want to be careful about that, as the default branch might have moved while you are working on your source branch.

Doing things like detecting being behind the default branch is important in those situations, as it might tell you if the temporary merge commit for the plan is more likely to have conflicts.