Provisioning a dask cluster on EC2 spot instances with Terraform

Download and install next HashiCorp tools

Create the base AMI with packer (install docker and build container):

packer build images/dask_base_cpu.json

Note: add argument --force to re-create an existing AMI.

Create the cluster with terraform:

terraform plan
terraform apply

Create SSH tunnel to access the dask scheduler:

ssh -N \
    -L 8786:localhost:8786 \
    -L 8787:localhost:8787 \
    ubuntu@[scheduler.public_ip]

Check the DASK scheduler UI: localhost:8787/status

Destroy all DASK cluster:

terraform destroy