How to check for worker health.

[lan-nguyen91]

Question: Worker HealthCheck

I have a simple implementation where API create jobs, and multiple workers just subscribe and process jobs.

I would like to scale the workers and monitor their health. Worker is just a simple script file and doesn't have any rest endpoint.
What would be the best way to get the health of the worker. Ideally worker will be running in a separate container which has no knowledge of the main API.

Is Bull provide such functionality, if not is it prefer if I just open another queue for healthcheck (sound like a hack) 🤔

Thanks.

[manast]

There is no health check built in. What we have is a getter that returns all the workers for a given queue https://github.com/OptimalBits/bull/blob/master/lib/worker.js#L18
I am afraid is not yet documented.

[hymloth]

It would be great if there was a way to have health check built in. My use case is that I run sandboxed workers on AWS Fargate, which after some hours seem like zombies (they seem to be up & running but they do not consume tasks). Why this happens is beyond the scope of this thread (perhaps a memory leak, a terrible bug etc) but if there was a health check functionality I could easily spot zombies and kill the corresponding tasks (and spawn new ones).

[manast]

@hymloth I will think about it. Although I cannot figure out an easy way to do it, maybe we could put the worker ID on every processed job and by examining the last N jobs we can see if there are workers that have not been processing any jobs and be reported as candidates for being Zombies or something like that.

[fmoessle]

Would be great to have a method to check if the queue still has an open connection to the database.

[joelrozen]

I am wanting the same as @fmoessle above. Just a bull method that returns the existing Redis.status. My use case is checking all my services are up. I can set a new connection to redis of course that tells me Redis is running, but doesn't necessarily tell me if Bull is still connected or not.

[manast]

You can use the "getWorkers" api to get all the current online workers that are connected to the redis instance.

[ajubin]

Is it possible to check health via await queue.client.ping(); ?

My need is a bit different though, I'm looking to implement a health-check system as in Nest, but there's nothing for bull

[manast]

@ajubin can you elaborate what do you mean with "health" in this context?

[ajubin]

Well, after some thought I'm not really sure how relevant it is.

my app use postgres with typeorm and a queue. It runs on a kubernetes cluster.

To check liveliness, i have an endpoint /healthz that returns a 200 if everything is ok.

I had a problem with my db clients which were stucked (no call to the db were made). But my service didn't restart as health check was ok.

My countermeasure: check on /healthz that my db is reachable.
As I use Bull also for queue, I was wondering if an healtcheck was necessary on redis client also.
After some thoughts, it seems not relevant to add this check as I'm not sure there are blocking connection with redis, or that the connection pool can be exhausted.

Is it clearer ?

[luanxuechao]

is there any update? I have the same issue.