Clarificaiton regarding concurrency controls

[salmonsteak1]

Hey, I'm trying to test out the duration parameter for the limits_concurrency feature. This is how I'm testing it:

# Test
class TestJob < ApplicationSolidqueueJob
  limits_concurrency to: 3, key: "test_job", duration: 2.seconds, group: "TestJob"

  # Infinite loop
  def perform
    sleep 1.second while true
  end
end

I will then run 20 of these jobs, with 10 solid queue workers. I've set concurrency_maintenance_interval to 1 second

From what I can observe, new jobs are being unblocked every 1 second when the dispatcher detects that a worker is running a job longer than the specifiedduration in limits_concurrency.

However, this means that eventually, all my workers will be hogged up by such jobs.

I know that it's possible to counter this by setting a job timeout using this approach, but I just wanted to clarify the usage of the duration value in limits_concurrency.

Given this current behavior, does this mean that this duration value in limits_concurrency refers to how much time the worker has to process this job? If it exceeds this duration, a new job will be unblocked (This is checked by the dispatcher every xxx interval, defined in concurrency_maintenance_interval). Nothing will be done for the worker that is still working on that job?

[rosa]

Nothing will be done for the worker that is still working on that job?

Yes, that's correct. duration is the time we can guarantee the concurrency controls to take effect. After that passes, jobs will be unlocked.

does this mean that this duration value in limits_concurrency refers to how much time the worker has to process this job?

It doesn't necessarily mean the time a job is running because it might still be waiting to be picked from the queue, for example, if workers are busy with other jobs.