Feature Request: include queues and worker index as arguments to on_worker_start hook

[Th3-M4jor]

For metrics reporting this would be helpful to know, much in the same way that Puma's on_worker_boot hook will receive the index of the current worker.

If such a feature would be accepted I may be able to work on a PR for it this weekend.

[rosa]

Oh, interesting! I've never used the index/key argument of Puma's on_worker_boot. What does it refer to exactly? What would it be here in Solid Queue? Thinking if it might make sense to pass itself 🤔

[Th3-M4jor]

Looking at the source code for Puma, I'm oversimplifying but it effectively does this:

worker_count.times do
  # find lowest index that's not in use by
  # a currently running worker
  index = next_worker_index

  fork { spawn_worker(index) }
end

So it'll try to reuse IDs whenever it has to refork.

In the case of SolidQueue, I'm thinking something similar?
A separate counter for each worker group and trying to re-use indexes if workers get restarted.

For example, given a worker config like this:

   workers:
    - queues: ["background"]
      threads: <%= ENV.fetch("SQ_BACKGROUND_JOB_THREADS", 3) %>
      processes: 1
      polling_interval: 0.1
    - queues: ["fast_jobs"]
      threads: <%= ENV.fetch("SQ_FAST_JOB_THREADS", 3) %>
      processes: 1
      polling_interval: 0.1
    - queues: ["slow_jobs"]
      threads: <%= ENV.fetch("SQ_SLOW_JOB_THREADS", 3) %>
      processes: 2
      polling_interval: 0.1

on_worker_start would then be called a total of 4 times with

• ["background"], 1
• ["fast_jobs"], 1
• ["slow_jobs"], 1
• ["slow_jobs"], 2

[rosa]

Solid Queue currently doesn't keep that kind of index per worker and workers don't know about their position within the list 🤔 Plus, the order is not really guaranteed, if they're registered and spawned in order, it's by chance. They do have a unique name that gets generated when they're registered, though. Maybe that could be used 🤔