Support dependencies from the find_in_parent_folders terragrunt function

[justin-watkinson-sp]

Hi there!

I was looking at using this to generate a config YAML that describes all of the relationships in my mono repo so that when a particular file changes, we can easily reason about which projects need to be re-built.

When I ran this against my test repo, I noticed that the certain files are left out of the generated YAML. For example, if my non-parent terragrunt.hcl contains this block:

locals {
  account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  region_vars  = read_terragrunt_config(find_in_parent_folders("region.hcl"))
}

The generated YAML doesn't seem to understand that these files exist and could be a dependency.

My question is - is that an expected behavior? Or perhaps could this be a feature that can be added to increase the scope of files it can detect and produce valid project YAML? I'd be happy to dive into the terragrunt parse code and see about adding something like this.

[dmattia]

Hello!

For right now, this is expected behavior as Terragrunt does not expose any api for working with the read_terragrunt_config.

Workaround

However, if you check through the https://github.com/transcend-io/terragrunt-atlantis-config#extra-dependencies section of this repo's README, I mention that function and show how you can add "account.hcl" and "region.hcl" into your dependencies, like so:

locals {
  extra_atlantis_dependencies = [
    find_in_parent_folders("account.hcl"),
    find_in_parent_folders("region.hcl"),
  ]
}

This should allow you to get full coverage of all dependencies in atlantis.yaml, but has the downside that it does require some extra markdown in your terragrunt.hcl files that use read_terragrunt_config.

read_terragrunt_config: a non-perfect function

So now, for why read_terragrunt_config makes me sad:

• I have an issue in Terragrunt asking about exposing info about the function here: Expose Dependencies from read_terragrunt_config calls in PartialDecodeSectionType gruntwork-io/terragrunt#1107
• That prompted a Terragrunt maintainer to make this follow up issue: read_terragrunt_config with dependency is broken gruntwork-io/terragrunt#1128
• The bug is significant enough that is one issue Terragrunt sort of recommended against using the function when you can: Only one level of includes is allowed gruntwork-io/terragrunt#303 (comment)

The core problem is that Terragrunt roughly parses files by saying:

1. Expand out all locals blocks
2. get all dependency blocks to construct the dependency graph

because read_terragrunt_config can be in locals, it is evaluated before Terragrunt looks at dependencies, meaning when I go to parse a file using Terragrunt's parser, there is no indication if that function was used or not, as the parsing takes place after locals are expanded out.

That being said, I also use that function a good amount in my monorepo and do use the extra_atlantis_dependencies override, so I'm not arguing that it shouldn't be used, just that there are some problems with that function.

Solving this issue fully

I think this is a solvable problem, but one that requires this repo to have its own parser, separate from using Terragrunt's. I actually already started doing this a bit in this file: https://github.com/transcend-io/terragrunt-atlantis-config/blob/master/cmd/parse_hcl.go#L36

You'll notice that when you're parsing raw HCL, that line lets you specify a set of functions you want to run over the config, which is how Terragrunt added in all their custom functions. I think we could create our own read_terragrunt_config function that writes out the filename (and any dependencies in that file) to a Set<string> in our parsing function, but that will be a significant amount of work

[justin-watkinson-sp]

Thanks for the detailed response, I'll read through the various issues you've cited and approach my team about the workaround.

We did start to go down the road of implementing a custom parser and have it is partially implemented for our use cases (Ruby, in our case). The main issue is that for every terragrunt feature, we have to reproduce the behaviors of locating, applying things in the correct order, etc., which while possible, doesn't seem like it will pay-off down the road.

[dmattia]

yeah, that makes sense. The reason I made this project in golang was because I specifically wanted to use Terragrunt's internal parsing logic when possible.

I will say, I ended up using extra_atlantis_dependencies quite a bit internally with our monorepo (~500 modules) to say things like "This module should be ran in CI anytime the AMI files change, or Dockerfile, or whatever". And it didn't take too long to search for usages of read_terragrunt_config and add them as locals. But if you have a much larger repo, I definitely see how this is a pain, and would love to eventually have this library work without any of the special locals metadata

[bbros-dev]

Thanks for all the effort you have put into this tools. We are exploring Terraform/Terragrunt/Atlantis.

The reproducible case we start with is the Terragrunt Infrastructure Live Example repo showing how to workaround the single include restriction.

Of course in that repository the root terragrunt.hcl uses read_terragrunt_config(find_in_parent_folders("account.hcl")), etc. Which led us here.

We find the suggested workaround does not resolve the issue:

locals {
  extra_atlantis_dependencies = [
    find_in_parent_folders("account.hcl"),
    find_in_parent_folders("region.hcl"),
  ]

  # Automatically load environment-level variables
  environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))
....

This still produces:

$ terragrunt-atlantis-config generate                                             
Error: Error in function call                                                                                
                                                    
  on /home/hedge/src/terragrunt-infrastructure-live-example/terragrunt.hcl line 14, in locals:                     14:   account_vars = read_terragrunt_config(find_in_parent_folders("account.hcl"))
                                                                                                                 Call to function "find_in_parent_folders" failed: ParentFileNotFound: Could not find a account.hcl in any of the
parent folders of /home/hedge/src/terragrunt-infrastructure-live-example/terragrunt.hcl. Cause: Traversed all the way to the root..

Appreciate any hints or tips that come to mind.

[dmattia]

Hello! Could you please confirm which terragrunt-atlantis-config version you are using @bbros-dev?

I just added a PR #74 that adds in the entire terragrunt-infrastructure-live-example repo as a test case and has it produce expected results. While I believe that. the latest version (v0.9.7) works, I am not positive that all previous versions would have worked as expected as a few of the latest patch releases have been bug fixes.

Could you please try the latest version? If that doesn't work, please check out the test example here https://github.com/transcend-io/terragrunt-atlantis-config/tree/master/test_examples/terragrunt-infrastructure-live-example and let me know what config you have that is different from that example (as I tried to copy what you have above, but also included a dependency on env.hcl

[bbros-dev]

Apologies for not including this information at first:

$ terragrunt-atlantis-config version
terragrunt-atlantis-config 0.9.7

I'll delete the repo and retry.....
Okay, have retried and we still see the error reported above. Interestingly terragrunt validate-all fails for us. We have reported that in terragrunt-infrastructure-live-example/issues/47.

Other version data.

$ terraform version
Terraform v0.13.3

$ terragrunt --version
terragrunt version v0.25.1

[dmattia]

ah! After looking through your issue on the other repo, I think I figured it out.

By default, terragrunt (and this tool) will think that any terragrunt.hcl file is a module you want to create resources for. But in your case (and how that repo recommends), there is a top level file that is only used as an import to other terragrunt files.

You have two options:

1. Add the "--ignore-parent-terragrunt" flag to this module, like we use in the test on the infrastructure-live repo: https://github.com/transcend-io/terragrunt-atlantis-config/pull/74/files#diff-012901cdeedad78e2c6f08d36a1ae826d6c7acfbd752ede87eb5bc00e6b5abc5R219
2. Rename the top level file to something like terragrunt-root.hcl.

Option 1 is nice if you want to keep the exact file layout, but option two should fix your issue with this library as well as with the validate-all command. Though I'm guessing they'll have a solution in that other repo for you, in which case I'd recommend option 1 for this library.