Ken does not start background indexation for some databases

[TestaDiRapa]

Description

I have a cluster of 6 nodes running CouchDB 3.4.2, with around 1000 databases of different sizes. Each database has between 20 and 40 design documents.

Recently, I updated the design documents on all databases and I let ken handle them in background, indexing them in the course of some days. Currently, there are no indexation tasks in progress (save the ones that are needed when documents are create/updated, but they are not a lot and very short).
The ken batch_channels configuration is set to 50 on all nodes.

The problem is that there is a subset of databases (~10%) where the indexation did not take place and ken is currently not picking up the task to index them in background.
If I query the view, then the indexation starts normally in foreground and complete successfully.

Steps to Reproduce

Create a lot of big databases, then update the design docs without triggering the foreground indexation.

Expected Behaviour

Ken should index all the design docs of all the databases in background.

Your Environment

• CouchDB version used: 3.4.2
• Browser name and version: not relevant
• Operating system and version: ubuntu 22.04 (couchdb compiled with dockerfile below)

FROM ubuntu:22.04

# Create app directory
WORKDIR /root

# Install dependencies
RUN apt-get update
RUN ln -fs /usr/share/zoneinfo/Europe/Brussels /etc/localtime
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN apt-get install -y gnupg2 wget curl vim git build-essential pkg-config libicu-dev libmozjs-78-dev libcurl4-openssl-dev libncurses-dev node-gyp npm libssl-dev help2man openjdk-21-jdk-headless
RUN git clone https://github.com/erlang/otp otp_src_27.1.2
WORKDIR /root/otp_src_27.1.2
RUN git checkout -b 27.1.2 44ffe8811dfcf3d2fe04d530c6e8fac5ca384e02
RUN bash -c 'export ERL_TOP=`pwd`; export LANG=C; ./configure; make; make release_tests; cd release/tests/test_server; /root/otp_src_27.1.2/bin/erl -s ts install -s ts smoke_test batch -s init stop; cd ..; tar zcvf otp-tests.tgz test_server; cd /root/otp_src_27.1.2; make install'
WORKDIR /root
RUN git clone https://github.com/apache/couchdb.git #9
WORKDIR /root/couchdb
RUN git checkout -b 3.4.2 6e5ad2a5c5479cb09722b4a7d13b3d59b7bb2a23
RUN bash -c 'curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash ; . /root/.nvm/nvm.sh; nvm install 18'
RUN ./configure  --disable-docs --spidermonkey-version 78 chdir=/media/data/src/couchdb
RUN bash -c '. /root/.nvm/nvm.sh; nvm use 18; make release'
WORKDIR /root/couchdb/rel
RUN tar zcvf couchdb.jammy-jellyfish.3.4.2.tgz couchdb

Additional Context