cryo: Allow GPU nodes to spawn across AZs

[yuvipanda]

We already do this for GCP, now we do it on AWS too.

Fixes #3334

EDIT: confirmed deployments

• 2i2c-aws-us
• carbonplan
• gridsst
• jupter-meets-the-earth
• nasa-cryo
• nasa-ghg
• nasa-veda
• openscapes
• smithsonian
• ubc-eoas
• victor

[yuvipanda]

Deployed and tested on cryo

[yuvipanda]

I'm going to terraform apply this across our AWS clusters

[consideRatio]

I think there may be an issue if we specify a region where the GPU isn't available, but I'm not sure - maybe not. Could also have been resolved with updates to cluster-autoscaler understanding it shouldn't attempt to scale up in a zone where the relevant GPU to attach isn't available.

[yuvipanda]

The terraform apply only creates EFS Mount Target in all zones, so nodes can spawn there if needed. Only cryocloud is updated now.

Was there a previous issue with cluster-autoscaler around this?

[damianavila]

I'm going to terraform apply this across our AWS clusters

@yuvipanda, can you confirm if this ⬆️ happened?

[consideRatio]

I did a sweep, this is now consistently applied across all AWS projects!