Exceeding cluster_max_cores error message

[droctothorpe]

Hello, Jim.

It may be helpful to clarify whether or not the resource usage of schedulers contributes to the cluster limits. For example, if cluster_max_cores = 10 and the scheduler uses 0.5 cores, can a user spin up 10 x 1-core workers or just 9? I think that the answer is 9 based on some testing that I conducted, but I didn't see anything in the documentation that clarified this distinction. I may have just missed it though. Happy to submit this documentation PR if it makes life easier for you.

When I tested the above scenario (cluster_max_cores = 2 and attempting to scale to 2 x 1-core workers), I ran into the following error message:

/opt/anaconda3/envs/dask/lib/python3.7/site-packages/dask_gateway/client.py:668: GatewayWarning: Scale request of 2 workers would exceed resource limit of 1 workers. Scaling to 1 instead.
  warnings.warn(GatewayWarning(msg["msg"]))

This message can be misleading because it gives the impression that the problem is with the cluster_max_workers configuration, when in fact, I had that set to 100. I suspect that exceeding the cluster_max_memory returns a similar error message. It might be worth making this error message more informative.

Curious to hear your thoughts.

Environment:
Gateway Server: 0.8.0
Gateway Client: 0.8.0
Python: 3.7.4
Dask: 2.22.0
Distributed: 2.22.0

[sebastian-luna-valero]

Hi,

Bumping into this myself just now. I do agree with what @droctothorpe is proposing above.

Thanks!