How can I increase parallelization in Gruntwork Pipelines jobs?

[zackproser]

A customer asked:

We're running several large jobs through Gruntwork Pipelines currently and would like to know if there's a way to increase the number of jobs run in parallel to lower the overall execution time?

---

Tracked in ticket #110219

[zackproser]

Our best recommendation for this currently is to modify your deploy-infra.sh script, which comes bundled as part of your Gruntwork Pipelines installation.

This script acts as the "glue" code between your version control system and the terragrunt plan and apply logic contained in the ECS Deploy Runner component of Gruntwork Pipelines.

Here's an example of the deploy-infra.sh script.

Within this script, we use the xargs unix utility to help format arguments to complex commands. xargs can also accept a -P flag, which stands for max-procs to increase the number of processes that xargs uses to execute a given command, as explained here.

For example, to have xargs spin up two processes, you would pass a value of 2 to the -P flag like so: xargs -P 2 <command>.

Find the right call to xargs

Note that the deploy.infra script has several different handler functions, each for a different workflow:

• handle_updated_folders
• handle_updated_envcommon
• handle_deleted_folders

Each handler ends with a call to xargs. Modify the appropriate xargs command in one or more handlers to pass the -P flag value of your choosing. Initially, we recommend trying a -P value of 2 to avoid excessive resource consumption:

xargs -P 2.