Pipelines plan_and_deploy fails with Cannot process module

[nicoStakelab]

Our pipeline started failing with the following error:

[ecs-deploy-runner][2023-01-25T21:41:58+0000] Terraform has been successfully initialized!
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 
[ecs-deploy-runner][2023-01-25T21:41:58+0000] You may now begin working with Terraform. Try running "terraform plan" to see
[ecs-deploy-runner][2023-01-25T21:41:58+0000] any changes that are required for your infrastructure. All Terraform commands
[ecs-deploy-runner][2023-01-25T21:41:58+0000] should now work.
[ecs-deploy-runner][2023-01-25T21:41:58+0000] If you ever set or change modules or backend configuration for Terraform,
[ecs-deploy-runner][2023-01-25T21:41:58+0000] rerun this command to reinitialize your working directory. If you forget, other
[ecs-deploy-runner][2023-01-25T21:41:58+0000] commands will detect it and remind you to do so if necessary.
[ecs-deploy-runner][2023-01-25T21:41:58+0000]  prefix=[/tmp/tmpucjppvg9/dev] 
[ecs-deploy-runner][2023-01-25T21:41:58+0000] time=2023-01-25T21:41:58Z level=error msg=15 errors occurred:
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/svc-protocols-crawler (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services, /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/redis, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/rds (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/openvpn-server, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/_global/account-baseline, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/openvpn-server, /tmp/tmpucjppvg9/dev/us-west-2/mgmt/ecs-deploy-runner, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]), finished with an error: Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/_global/account-baseline, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/openvpn-server, /tmp/tmpucjppvg9/dev/us-west-2/mgmt/ecs-deploy-runner, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/openvpn-server (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-metrics-server (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/_global/route53-public, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/rds, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace]), finished with an error: Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/_global/route53-public, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/rds, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/route53-private (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/_global/account-baseline, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/openvpn-server, /tmp/tmpucjppvg9/dev/us-west-2/mgmt/ecs-deploy-runner, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/_global/route53-public, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/rds, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/api-core (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services, /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/redis, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/services/stakelab-dashboard (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-cluster, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-applications-namespace, /tmp/tmpucjppvg9/dev/us-west-2/dev/services/eks-core-services, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/mgmt/ecs-deploy-runner (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/mgmt/networking/vpc]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/mgmt/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] 	* Cannot process module Module /tmp/tmpucjppvg9/dev/us-west-2/dev/data-stores/redis (excluded: false, assume applied: false, dependencies: [/tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc, /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/openvpn-server, /tmp/tmpucjppvg9/dev/us-west-2/_regional/sns-topic]) because one of its dependencies, Module /tmp/tmpucjppvg9/dev/us-west-2/dev/networking/vpc (excluded: false, assume applied: false, dependencies: []), finished with an error: exit status 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] [INFO] [infrastructure-deploy-script] 2023-01-25 21:41:58  "terragrunt plan-all" exited with code 126
[ecs-deploy-runner][2023-01-25T21:41:58+0000] Error: Could not run "terragrunt plan-all"
[ecs-deploy-runner][2023-01-25T21:41:58+0000] exit status 1
[ecs-deploy-runner][2023-01-25T21:41:58+0000] ERROR: exit status 1
ERROR: Deployment failed
xargs: bash: exited with status 255; aborting
Error: Process completed with exit code 124.

Has anybody seen this issue before? We are able to run it locally, just the pipeline seems to be failing.

---

Tracked in ticket #109836

[gitsstewart]

@nicoStakelab have you made any changes to the VPC module that is referenced in the error message or to any configurations? Those would be helpful to see for troubleshooting. This could be anything from a terraform issue to a configuration that AWS or some other service no longer supports.

[denis256]

Hello,
looks like it is failing one of the bash scripts used in deployment:

ERROR: Deployment failed
xargs: bash: exited with status 255; aborting
Error: Process completed with exit code 124.

Can be attempted to enable debug logs or defined "set -x" in the script to see where it fails

[antonl]

We have been having the same issues; it seems to be some sort of concurrency issue, but the IaC deployer currently doesn't allow limiting concurrency.

[nicoStakelab]

Interesting. We updated the vpc terraform version to 0.100 and that seemed to resolve the issue:

terraform {
  source = "${include.envcommon.locals.source_base_url}?ref=v0.100.0"
}

[gitsstewart]

Thanks for the update @nicoStakelab