Management UI is not responsible when server is under (heavy) load

[dario-l]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.5

Erlang version used

27.2.x

Operating system (distribution) used

Ubuntu 24.04

How is RabbitMQ deployed?

Generic binary package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

.

Steps to reproduce the behavior in question

.

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

We have not a small modular monolithic tenanted application which uses ~1200 queues. Each tenant have own instance which is deployed on two application servers. Each tenant have own virtual host and each instance opens two connections (one for publishing and one for consumers). Each consumer have own channel, it means app opens ~1200 channels on consumers connection.

Rabbit is a cluster working on 3 nodes.

For now we have 7 tenants installed... so Rabbit manage 14 connections and ~20k channels.

When one tenant is under load (app processing around 3k messages per second) the UI management stays not responsible.

PS
We want to run more tenant instances. Could such a large number of channels be the reason for this?

[michaelklishin]

@dario-l your cluster is deprived of CPU resources, and/or there are apps that invoke GET /api/queues, GET /api/channels and such without pagination.

Every channel, even if it is not active otherwise, emits stats every 5 seconds. Which is why the limit on the maximum number of channels per connection has been going down gradually every couple of years.

Shared clusters will inevitably run into the noisy neighbor problem (someone else's apps hogging most of CPU or other resources, including network bandwidth). All the tuning imaginable can only delay the inevitable.

Which is why VMware products provision separate clusters by default and only leave the shared cluster option as an opt-in feature recommended only for integration testing and such.

It's easy to see that RabbitMQ-as-a-Service providers severely limit the number of connections
and channels, and often message rates, on shared plans.

[michaelklishin]

Also, the Virtual hosts guide explicitly states that they only offer logical separation and offer nothing in terms of physical resource separation.

So, the pool of resources (cores, disk space, disk I/O, network bandwidth) in a cluster must be adequate for all tenants or you have no choice but to use separate clusters.

[dario-l]

Thanks for the clarification. As for resources, we are aware of all that.

Every channel, even if it is not active otherwise, emits stats every 5 seconds. Which is why the limit on the maximum number of channels per connection has been going down gradually every couple of years.

This sentence I think is most important for us. Definitely we must refactor to decrease number of channels.

Our try is to make only one channel per module. For now we have about 60 modules. This will reduce channels drastically.

[michaelklishin]

That would help for a while and follows the general recommendation of using a relatively small number of long-lived channels.

Not necessarily one per application instance but few applications really need > 10 channels.
The key is to not consider channels to be free in terms of resources.