errors block doesn't seem to be used for output lookup

[MichaelFoleyFZ]

Describe the bug

We are currently using terraform cloud to manage state storage and when we need to setup new set of units with dependencies we get an error when the outputs are looked up

12:47:40.725 DEBUG  [log_stores] Running command: terraform output -json
12:47:40.725 DEBUG  [log_stores] Engine is not enabled, running command directly in ./log_stores/.terragrunt-cache/ZRBfJ-EMc3VqKO0sG5FDLG_h7nQ/WGYDzM3b26Ekahx2CNhbilsxIOw
╷
│ Error: could not read state version outputs: resource not found
│ 
│ 

This makes sense since we don't have terraform cloud workspace state created yet, so I added mock outputs in to our dependency check to hopefully get around it

dependency "log_stores" {
  config_path                             = "../log_stores"
  mock_outputs_allowed_terraform_commands = ["plan", "validate"]
  mock_outputs = {
    kms_key_id = "arn:aws:kms:ap-southeast-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
  }
}

But terragrunt still fails since its running terraform output -json before using the mock outputs. skip_outputs was an option but creates issues with when it should be enabled or disabled (i.e. when the terraform workspace exists it shouldn't skip )

So I tried the new errors block.

errors {
  ignore "state_missing" {
    ignorable_errors = [
      ".*Error: could not read state version outputs.*"
    ]
    message = "Ignoring missing state for new stack"
  }
}

But it looks like the errors block doesn't get evaluated when running the output look ups. Since it still fails and doesn't produce anything to say that the error has been ignored.

Steps To Reproduce

terragrunt.hcl

generate "terraform" {
  path      = "terraform.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<-EOT
    terraform {
      cloud {
        organization = "<insert tf cloud org here>"

        workspaces {
          name = "ws-${replace(path_relative_to_include(), "/", "-")}"
        }
      }
    }
  EOT
}

errors {
  ignore "state_missing" {
    ignorable_errors = [
      ".*Error: could not read state version outputs.*"
    ]
    message = "Ignoring missing state for new stack"
  }
}

unit1/terragrunt.hcl

terraform {
  source = "main.tf"
}

include "root" {
  path = find_in_parent_folders()
}

unit1/main.tf

resource "terraform_data" "data" {
  input = "abc123"
}

output "data" {
  value = terraform_data.data.output
}

unit2/terragrunt.hcl

terraform {
  source = "main.tf"
}

dependency "unit1" {
  config_path                             = "../unit1"
  mock_outputs_allowed_terraform_commands = ["plan", "validate"]
  mock_outputs = {
    data = "def456"
  }
}

include "root" {
  path = find_in_parent_folders()
}

unit2/main.tf

variable "unit_a_data" {
  type = string
}

resource "terraform_data" "unit_a_data" {
  input = var.unit_a_data
}

output "data" {
  value = terraform_data.unit_a_data.output
}

Then at the root of this workspace run

terragrunt run-all plan

13:00:34.191 STDOUT [unit1] terraform: again to reinitialize your working directory.
╷
│ Error: could not read state version outputs: resource not found
│ 
│ 
╵
13:00:37.047 ERROR  [unit1] terraform invocation failed in ./unit1/.terragrunt-cache/FBQW8Cc6xOlFLI-6v8PSRgS_iuI/nemD4fDbQ4HJKvm5JdK4t5BRPdU
13:00:37.047 ERROR  [unit2] Module ./unit2 has finished with an error
13:00:37.048 ERROR  error occurred:

* Failed to execute "terraform output -json" in ./unit1/.terragrunt-cache/FBQW8Cc6xOlFLI-6v8PSRgS_iuI/nemD4fDbQ4HJKvm5JdK4t5BRPdU
  ╷
  │ Error: could not read state version outputs: resource not found
  │ 
  │ 
  ╵
  
  exit status 1

Expected behavior

When running the plan step the outputs are attempted to be looked up and if the terraform workspace doesn't have outputs the mock outputs are returned because the error Error: could not read state version outputs: resource not found was ignored in the errors block

Nice to haves

• Terminal output
• Screenshots

Versions

• Terragrunt version: 0.69.13
• OpenTofu/Terraform version: terraform v1.9.8
• Environment details (Ubuntu 20.04, Windows 10, etc.): MacOS 15.1.1

Additional context

This might quite be the right approach as a bug, but the errors block feels like the right spot for it to be handled.

[yhakbar]

Hey @MichaelFoleyFZ ,

I don't have a Terraform Cloud account, so I can't reproduce your issue, exactly.

In a setup like this:

$ fd -tf -x bash -c 'echo "# {}" && cat {} && echo'
# ./unit2/terragrunt.hcl
dependency "unit1" {
  config_path                             = "../unit1"
  mock_outputs_allowed_terraform_commands = ["plan", "validate"]
  mock_outputs = {
    data = "def456"
  }
}

inputs = {
        unit_a_data = dependency.unit1.outputs.data
}

# ./unit1/main.tf
resource "terraform_data" "data" {
  input = "abc123"
}

output "data" {
  value = terraform_data.data.output
}

# ./unit1/terragrunt.hcl

# ./unit2/main.tf
variable "unit_a_data" {
  type = string
}

resource "terraform_data" "unit_a_data" {
  input = var.unit_a_data
}

output "data" {
  value = terraform_data.unit_a_data.output
}

I'm able to do a run-all plan at the root of the repository without issue. I do see the following when I run terragrunt output -json in the unit1 directory, which I suspect you might not:

$ terragrunt output -json
10:49:14.346 STDOUT tofu: Initializing the backend...
10:49:14.347 STDOUT tofu: Initializing provider plugins...
10:49:14.347 STDOUT tofu: - terraform.io/builtin/terraform is built in to OpenTofu
10:49:14.348 STDOUT tofu: OpenTofu has been successfully initialized!
10:49:14.348 STDOUT tofu:
10:49:14.348 STDOUT tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
10:49:14.348 STDOUT tofu: any changes that are required for your infrastructure. All OpenTofu commands
10:49:14.348 STDOUT tofu: should now work.
10:49:14.348 STDOUT tofu: If you ever set or change modules or backend configuration for OpenTofu,
10:49:14.348 STDOUT tofu: rerun this command to reinitialize your working directory. If you forget, other
10:49:14.348 STDOUT tofu: commands will detect it and remind you to do so if necessary.
{}

You can see there that an empty JSON object is returned even though I haven't applied yet. Terragrunt uses that to determine that it should be mocking inputs from another unit. Note that in this scenario, Terragrunt hasn't encountered any error running output from the dependency. It just sees that outputs are empty.

What do you see when you try to run output -json in unit1?

[MichaelFoleyFZ]

Looks like terraform cloud fails to return an empty object and errors instead

terragrunt --terragrunt-log-level debug output -json  
11:44:05.231 DEBUG  Terragrunt Version: 0.69.13
11:44:05.235 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.237 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.237 DEBUG  [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
11:44:05.237 DEBUG  Running command: terraform --version
11:44:05.237 DEBUG  Engine is not enabled, running command directly in .
11:44:05.442 DEBUG  terraform version: 1.9.8
11:44:05.442 DEBUG  Reading Terragrunt config file at ./terragrunt.hcl
11:44:05.442 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.443 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.444 DEBUG  [Partial] Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
11:44:05.444 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.445 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.445 DEBUG  Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
11:44:05.446 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.446 DEBUG  Did not find any locals block: skipping evaluation.
11:44:05.446 DEBUG  Included config ../terragrunt.hcl has strategy shallow merge: merging config in (shallow).
11:44:05.447 DEBUG  terraform files in ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4 are up to date. Will not download again.
11:44:05.447 DEBUG  Copying files from . into ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4
11:44:05.448 DEBUG  Setting working directory to ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4
11:44:05.448 DEBUG  The file path ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4/terraform.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.
11:44:05.448 DEBUG  Generated file ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4/terraform.tf.
11:44:05.449 DEBUG  Running command: terraform init
11:44:05.449 DEBUG  Engine is not enabled, running command directly in ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4
11:44:05.493 STDOUT terraform: Initializing HCP Terraform...
11:44:09.388 STDOUT terraform: Initializing provider plugins...
11:44:09.389 STDOUT terraform: - terraform.io/builtin/terraform is built in to Terraform
11:44:09.389 STDOUT terraform: HCP Terraform has been successfully initialized!
11:44:09.389 STDOUT terraform: 
11:44:09.389 STDOUT terraform: You may now begin working with HCP Terraform. Try running "terraform plan" to
11:44:09.389 STDOUT terraform: see any changes that are required for your infrastructure.
11:44:09.389 STDOUT terraform: If you ever set or change modules or Terraform Settings, run "terraform init"
11:44:09.389 STDOUT terraform: again to reinitialize your working directory.
11:44:09.390 DEBUG  Running command: terraform output -json
11:44:09.390 DEBUG  Engine is not enabled, running command directly in ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4
╷
│ Error: could not read state version outputs: resource not found
│ 
│ 
╵
11:44:13.384 ERROR  terraform invocation failed in ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4
11:44:13.384 ERROR  error processing error handling rules: error occurred:

* Failed to execute "terraform output -json" in ./.terragrunt-cache/9MEgIme96jJYosRHgwDAVTzkLeY/iHekrlesyyo2BrbT6WZ3yAy62g4
  ╷
  │ Error: could not read state version outputs: resource not found
  │ 
  │ 
  ╵
  exit status 1

Terraform cloud seems to only return outputs if you have run at least one apply against the cloud., other wise it fails on the output return

[yhakbar]

To close the loop on this, I'm not sure the errors block should do anything to help with this edge case.

Other providers don't encounter this issue, and I'm inclined to say it's a bug in the Terraform cloud remote backend, as it breaks the pattern established by other remote backends. I wouldn't want dependencies to treat empty stdout with stderr content as the same thing as an empty json object, as that would indicate a problem with the backend for everything else.

For now, I think your options are to:

1. Use a different backend.
2. Only plan units without unapplied dependencies.
3. Find some way to apply empty configurations using a before_hook so that you get a valid response from output.

[MichaelFoleyFZ]

Yeah totally fair call, it felt like a bit of a cludge in using errors for something that the user has no control over with the outputs lookup.
We will probably be looking at using a different backend in the future, so might just look at a before_hook as a temp workaround, thanks for looking into it.