High Redis CPU on high number of waiting jobs worked on

[derN3rd]

Hey there,

We are experiencing a huge CPU increase if we have a lot of jobs in waiting state and try to process them fast.
Memory is not an issue, but job execution time ramps up quite fast to a point where the whole queue + redis locks up basically. (We are using jobs with removeOnFinish: false)

We are still using bull 3.15.9 for some deployment reasons and I'm now trying to figure out if this was fixed in a later version of bull or is still a huge performance bottleneck? But looking from the changes there don't seem to be changes >3.15.9 regarding this performance issue.

Thanks in advance

As seen in the pic below, once we start to pile up with waiting jobs, all processors take much longer to process jobs, as well as adding jobs takes longer. We can see that at 20:03 the Redis CPU is at 99% and everything starts to degrade.

[manast]

Are you using rate limit by any chance?

[derN3rd]

No. Just a normal queue with some delayed jobs.

We also tried not using any delayed jobs at all and just adding jobs in batches via cronjob at different times, but again as soon as we hit somewhere between 50-100k jobs in waiting, the Redis was locking up at 99% CPU

Our setup for this queue is:

• ElastiCache cache.m5.xlarge (Redis 7.1)
• Bull 3.15.9 on all nodes (where jobs are added and consumed)
• Queue Setup with

   new BullQueue('pushNotificationRetention', {
     redis: { db: 0, host },
   })

• Job adding via

   queue.add(
     'sendSummary',
     {
       userId,
     },
     {
       removeOnComplete: false,
       jobId: `${userId}-${schedule}`,
       delay: delayInMs,
     }
   )

• Job consumer is setup as:

   queue.process('pushNotificationRetention', 50, async (job: Job): Promise<void> => {
       if (remoteConfig.skipPushRetentionJobs) {
         return
       }
   
       await sendReactivationalPushForUser({
         ...job.data,
       })
     }
   )

[derN3rd]

@manast Could this be related to how we handle deletion of jobs?:

As seen, we use removeOnComplete: false in the job creation and have a separate process listening on queue.on('global:completed') to capture some analytics and then call jobs.remove().
We need to look into the job.data, therefore we cannot use removeOnComplete:true as we only get the job ID in the event afaik.

Would periodically cleaning the jobs by using queue.clean('completed') resolve this issue? Or do you even see other issues that could arise from this?

Thanks in advance

[manast]

I recall we had a performance issue when deleting many jobs, but not one by one. You could issue a Redis MONITOR command and then see which commands are being issued when running your custom delete code, then you will be able to see if it is the cause or not.