adapt: could be added to cluster options?

[shaunc]

I would like to expose adaptation as a cluster option (in my case for kubernetes deployment, but probably would be good for any). (active, minimum, maximum)

Is there any technical reason why adaptation has been kept separate from other cluster settings? Especially since I will be creating clusters via prefect agent, a separate call to adapt() seems problematic.

[jcrist]

Adapt arguments aren't configured in the cluster constructor for any other dask cluster classes - it's not part of the standard distributed.deploy.Cluster insterface.

In Prefect we've already thought about this, you can pass adapt_kwargs to DaskExecutor to configure the arguments to pass to an adapt call after cluster creation:

executor = DaskExecutor(
    cluster_class="dask_gateway.GatewayCluster",
    cluster_kwargs={...},
    adapt_kwargs={"maximum": 10}
)

Is this not sufficient for your needs?

[shaunc]

Hmm... maybe adaptation options should be made part of dask.distributed? :) Seems odd to separate these off. I am hoping to specify clusters by "profile" (and maybe by user) as you explain in the doc, and want to have adaptation settings as an integral part of a profile.

Perhaps (if its not feasible to put these into dask.distributed) there is a way to setup an "on-cluster-create" hook which could be specified in cooperation with the option handler in the helm deployment: the handler could register a callback that set the adaptation options?

[jcrist]

Things in the cluster constructor (and cluster_options) are static configuration options for an individual cluster, they define static properties of a cluster that aren't updated at any point later. In contrast, adapt/scale settings are dynamic, the settings configured by an individual adapt or scale call can be changed later by the user. I think trying to enforce these through configuration on the dask-gateway side is errant and out of scope for this project.

Note that there are static, per cluster resource limits you can set, so if your goal is to limit user resources to certain bounds you could make use of these instead https://gateway.dask.org/resource-limits.html. adapt_kwargs could then be set to adapt_kwargs={"maximum": None} for all users, with the true limit configured by the resource limits.

Alternatively, we could make the adapt_kwargs configurable on the prefect side as part of the prefect config file, then you could specify these as part of your prefect images (or through environment variables).

I guess I'm looking for how the current configuration options make things harder for you. What is your goal here (the what), rather than the mechanism you'd like to do it with (the how)?

[shaunc]

The "what" is that I'd like to have defaults with enforceable bounds on "what sort of cluster a given profile gets" and have it work by default without too much thought on the part of a user, who hopefully will be worrying about other things (like cleaning data, etc). Although scale and adaptivity are configurable dynamically, I'd like clusters to adapt by default within the bounds set by the profile. This seems like the most intuitive setting for a user who hasn't profiled their tasks yet, and/or is just developing them. I would think that this would be useful for any dask_gateway administrator to be able to set up for their users, independent of whether they are using prefect.

Edit -- it may be that adapt_kwargs={"maximum": None} is all I need. Didn't realize that that was possible.

[shaunc]

I don't know if this is the right way to think about it, but I'm seeing adaptivity in analogy to the k8 cluster autoscaler. CA is a bit fiddly to setup, but usually when I deploy an application, I don't have to think about it. Of course the analogy is inexact, because k8 is global wrt to things running in the cluster, while a dask cluster is specific to the flow (though having a pool of dask clusters to run flows on is something else I may want at some point...). Still it seems very plausible that users should have to think or tune cluster size as little as possible by default (within sensible, cost-effective limits), and adaptivity seems the best way to get them this.