Request processing thread of netty was full, triggering the rejection strategy

[liuyi-stack]

024-09-30 15:29:53 default [lettuce-timer-10-1] WARN io.netty.util.HashedWheelTimer - An exception was thrown by TimerTask.
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:125)
at io.netty.util.concurrent.AbstractEventExecutorGroup.submit(AbstractEventExecutorGroup.java:45)
at io.lettuce.core.protocol.ConnectionWatchdog.lambda$scheduleReconnect$3(ConnectionWatchdog.java:232)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-09-30 15:29:53 default [lettuce-epollEventLoop-11-3] WARN i.l.core.protocol.CommandHandler - null Unexpected exception during request: java.util.concurrent.RejectedExecutionException: event executor terminated
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
at io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:261)
at io.netty.util.concurrent.AbstractScheduledEventExecutor.schedule(AbstractScheduledEventExecutor.java:177)
at io.netty.util.concurrent.AbstractEventExecutorGroup.schedule(AbstractEventExecutorGroup.java:50)
at io.netty.util.concurrent.AbstractEventExecutorGroup.schedule(AbstractEventExecutorGroup.java:32)
at io.lettuce.core.protocol.CommandExpiryWriter.potentiallyExpire(CommandExpiryWriter.java:164)
at io.lettuce.core.protocol.CommandExpiryWriter.write(CommandExpiryWriter.java:111)
at io.lettuce.core.RedisChannelHandler.dispatch(RedisChannelHandler.java:187)
at io.lettuce.core.StatefulRedisConnectionImpl.dispatch(StatefulRedisConnectionImpl.java:152)
at io.lettuce.core.AbstractRedisAsyncCommands.dispatch(AbstractRedisAsyncCommands.java:467)
at io.lettuce.core.AbstractRedisAsyncCommands.authAsync(AbstractRedisAsyncCommands.java:86)
at io.lettuce.core.StatefulRedisConnectionImpl.activated(StatefulRedisConnectionImpl.java:130)
at io.lettuce.core.protocol.DefaultEndpoint.lambda$notifyChannelActive$0(DefaultEndpoint.java:379)
at io.lettuce.core.protocol.SharedLock.lambda$doExclusive$0(SharedLock.java:82)
at io.lettuce.core.protocol.SharedLock.doExclusive(SharedLock.java:103)
at io.lettuce.core.protocol.SharedLock.doExclusive(SharedLock.java:81)
at io.lettuce.core.protocol.DefaultEndpoint.notifyChannelActive(DefaultEndpoint.java:363)
at io.lettuce.core.protocol.CommandHandler.channelActive(CommandHandler.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:214)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:207)
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:69)
at io.lettuce.core.ChannelGroupListener.channelActive(ChannelGroupListener.java:40)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:214)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:207)
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:69)
at io.lettuce.core.PlainChannelInitializer$1.channelActive(PlainChannelInitializer.java:109)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:214)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:207)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:214)
at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:620)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:653)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)

[tishun]

Hey @liuyi-stack,
unfortunately we only could communicate in English, so please try to address your question that way.

Using Google Translate seems to translate that

The backend springboot service in the online production environment suddenly crashed, and all requests could not be accessed. After downloading the exception log file, the heap exception was analyzed and it seemed that the request processing thread of netty was full, triggering the rejection strategy

There is very little information to provide any meaningful advice. Seems like the event executor group gets terminated. This happens typically on application shutdown. Without further details it would be hard to understand why.

[liuyi-stack]

This problem does not recur often, and it also appeared suddenly before. The client cannot request the server. After checking the log, I found the following error log:
2024-09-30 15:29:53 default [lettuce-epollEventLoop-11-3] WARN i.l.core.protocol.CommandHandler - null Unexpected exception during request: java.util.concurrent.RejectedExecutionException: event executor terminated
java.util.concurrent.RejectedExecutionException: event executor terminated

[tishun]

event executor terminated

See #1773 (for example)

RejectedExecutionException is a consequence of shutting down the EventLoop (ClientResources.shutdown(), RedisClient.shutdown()). Please ensure that you close the event loop after completing all outstanding Redis commands.