Set semaphore to 2 for "request_task" API

[ppigazzini]

Tested working fine with a fleet of 75k cores 10k machines,
Waitress configured with threads=4 and connection_limit=100
HW usage: RAM 72%, primary Waitress instance CPU 85-95%

This is a good compromise for threads "privileged" for "request_task" API, and threads free for any other APIs, while keeping Waitress threads=4.

Raising the number of Waitress threads uses more RAM and can requires lowering the MongoDB cache size to avoid the OOM killer.

see #2003

[xoto10]

Nice :)

Do we have any information on how long the request_task calls take if rejected by the semaphore?

I assume it's very fast, but it helps to theorise about what's happening if we had a rough idea of the real number.

[vondele]

Do we have any information on how long the request_task calls take if rejected by the semaphore?

handled in 370.54ms (server: 210.34ms)
handled in 319.99ms (server: 151.06ms)
handled in 339.38ms (server: 158.88ms)
handled in 280.25ms (server: 52.08ms)
handled in 235.64ms (server: 1.82ms)
handled in 446.53ms (server: 1.25ms)
handled in 484.77ms (server: 8.73ms)
handled in 798.18ms (server: 138.25ms)
handled in 292.24ms (server: 119.02ms)
handled in 376.86ms (server: 0.72ms)
handled in 1307.80ms (server: 5.86ms)
handled in 406.99ms (server: 1.32ms)
handled in 155.74ms (server: 6.02ms)
handled in 551.27ms (server: 278.69ms)
handled in 405.88ms (server: 0.93ms)
handled in 534.28ms (server: 253.10ms)

so probably 1ms on reject, 150ms on average when passing