Support for AWS ECS Backend

[AndrewHannigan]

Requesting support for AWS ECS with Fargate support. A previous issue was opened with this same request, but closed in favor of Dask CloudProvider. I require Dask Gateway so admins can ultimately control allocation of resource upon request from users, but we like to run everything on ECS with Fargate. ECS support would really streamline the deployment of Dask Gateway for us, as it would not introduce maintenance of a Kubernetes/Hadoop cluster into the mix while still letting admins keep control of resource allocation.

[kinghuang]

Although I closed my request in #317, I'm still interested in having this in the gateway, too.

[jcrist]

I think this would be a fine thing to support, but I'm not likely to work on this myself in the near future. For reference on how one might go about adding this in a PR, see my comment here.

[AndrewHannigan]

Gotcha. @jcrist Does dask gateway use dask cloudprovider at all under the hood for provisioning clusters dynamically? Or did the cloud provider project come after the gateway project? All the functionality that I would need is definitely available in the dask cloudprovider, but I noticed your comments on what a PR might look like didn't mention dask cloudprovider. Just curious if all the work that went into cloudprovider can be leveraged by the gateway.

[jcrist]

Due to the requirements of dask-gateway (DB, resilient to failure, multi-user, scalability issues, etc...), we have to re-implement all backends and don't make use of other dask cluster-managers. In practice this isn't too bad. See #153 (comment) for more info.

[jacobtomlinson]

I'm still disappointed this is the case. Given all the work that goes into packages like dask-cloudprovider it's a shame that effort can't be leveraged here.

I would be very interested in reopening the conversation around how we can better align things to build reusable components.

[joncourt]

There is a ServerDisconnected error issue also with a Fargate based EKS deployment... is that related to this issue or should I raise a new one?

[jacobtomlinson]

That is not related to this issue. Here we are discussing a pure ECS backend implementation, not Kubernetes.

@joncourt If you could raise a new issue that would be great!