Large number of requests waiting on build

[rgaiacs]

@arnim contacted me because GESIS server has a large (>100) number of requests waiting on build step as indicated in Graphana

Screenshot source: https://grafana.mybinder.org/d/3SpLQinmk/1-overview?orgId=1&var-cluster=000000002&from=1700682034622&to=1703274034623&viewPanel=34

A similar large (>200) number of requests waiting on build step is observed on OVH:

Screenshot source: https://grafana.mybinder.org/d/3SpLQinmk/1-overview?orgId=1&var-cluster=000000004&from=1700682278831&to=1703274278832&viewPanel=34

And Curvenote:

Screenshot source: https://grafana.mybinder.org/d/3SpLQinmk/1-overview?orgId=1&var-cluster=b67087d4-25bd-41aa-8ecd-940252a85e24&from=1700682335597&to=1703274335599&viewPanel=34

I suspect that #2857 introduced some change that is causing this.

[rgaiacs]

The builds counter was reset https://grafana.mybinder.org/d/3SpLQinmk/1-overview?orgId=1&var-cluster=000000004&from=1709017624637&to=1709622424638&viewPanel=34

Maybe related to #2937

[manics]

Or it might be #2938 which includes jupyterhub/binderhub#1828
Can you check the dind logs for Removing incorrect socket directory {{ $hostSocketPath }}?

[rgaiacs]

This is old and I cannot look at the logs.