Extend documentation of concurrency controls

[cedricpim]

Hello 👋

I had a usage for concurrency controls and I was investigating how it worked and I found two pieces of information that I think could be useful in the README section because they weren't immediately clear to me.

The first piece was that the dispatcher, only releases one job with duration expired per run (so, using the default value of 5 minutes for concurrency_maintenance_interval, it means only one expired job is released every 5 minutes).

The second piece of information was that, on a job (with concurrency controls) completion, the duration of the already blocked jobs is "refreshed" (from what I was able to see, this happens because the concurrency keys to be considered for release must either have an open semaphore or no semaphore at all and, since dispatcher destroys expired semaphores, when a job finishes, the semaphore expiration time is increased).

Let me know if you want me to prepare a PR or you prefer to write this on your own words (or if I am wrong of course).

[rosa]

Hey @cedricpim, sure! Feel free to open a PR with some changes to the README.

The first piece was that the dispatcher, only releases one job with duration expired per run

One per concurrency key, to ensure concurrency limits are still respected for the released jobs. In theory, this shouldn't need to release any jobs unless you have a job that is very long, as jobs should release the following blocked jobs when they finish, but of course, there's a race condition here: #456. This means the dispatcher's concurrency maintenance fallback is necessary in that case until I fix that 😳 (it's a tricky one).

on a job (with concurrency controls) completion, the duration of the already blocked jobs is "refreshed"

Do you mean if a job finishes and un-blocks another job, then the new job renews the semaphore expiry time? Then yes, that's correct 👍