Unexpectedly high memory usage per channel after pileup

[BalintHarmatAtBetssonGroup]

We have noticed that following larger pileups, the memory consumed per channel grows, and doesn't ever go down until node restart. For example, one of our clusters has ~4300 total channels, with ~200Mib memory consumption according to the node memory details. After pileup, with the same number of channels, the memory consumption goes up to 9Gib, a ~40x increase, and stays there even after all queues have been cleared, taking up by far the most memory on the node, that reports a total Runtime Used memory of 11 Gib.

We do pool channels, no connection uses more than 63, most less than 10, we just have a lot of services connected.

Is this a known issue? or a non-issue and will be reclaimed at the last minute if needed? So far we usually resort to a rolling-restart during slow hours so we don't have to find out at when the next pileup happens.

Specifically on RabbitMQ 3.8.9

[lukebakken]

Hello, thanks for using RabbitMQ.

We can't assist you unless we know more about your environment -

• RabbitMQ version (provided, 3.8.9)
• Erlang version
• Operating system / virtualization used to run RabbitMQ (with version)
• Queue type being used (quorum vs classic, are classic queues mirrored?)

What does "larger pileups" mean? Do you mean you have many messages in the "Ready" state?

Finally, we can't spend time assisting users who are on an unsupported version of RabbitMQ. RabbitMQ 3.8.9 has been out of support for quite a while now.

https://www.rabbitmq.com/release-information

[BalintHarmatAtBetssonGroup]

Hello, thanks for the quick reponse, we are trying to arrange for a cluster upgrade, but in the meantime I wanted to ask if this is some known issue that needs another solution / usage pattern.

Will report back if the upgrade happens and solves it or not.

To give more context that will be valid even after upgrade, pileup in this case means ~1 million messages in Ready state.

[lukebakken]

There are several questions I asked, be sure to provide that information if / when you follow-up. Thanks!

[mkuratczyk]

We certainly don't have enough data to know what the problem is but if you use TLS, it could be #9761.

[michaelklishin]

@BalintHarmatAtBetssonGroup RabbitMQ 3.8 has long reached EOL. Start with collecting relevant data instead of guessing. The docs explain where to start.

For a large number of connections, see this section, otherwise only having good monitoring in place will allow you to make an informed decision.

Obviously we recommend upgrading to 3.13.x, currently 3.13.1.

[jignesh-sen]

Hi,
we are facing similar issue in our environment, Here;s the information if anyone could help solve the problem of high channel count
RabbitMQ version (provided, 4.0.2)
Erlang version 26.2.5
Operating system / virtualization used to run RabbitMQ (with version) - Win server 2016 Version 1607
Queue type being used (quorum vs classic, are classic queues mirrored?) quorum queues being used as classic one was removed

So the problem is whenever we do a server reboot we get a high channel count of atleast 12K and it stops accepting connections and we are not able to launch any secrets. what is the best solution to fix the issue with high channel count.

[michaelklishin]

@jignesh-sen RabbitMQ 4.0.x is out of community support.

The best solution to fix an issue is to understand it, and we don't have any data to even make an educated guess. However, we do know that except for shovels and federation links, both of which are ultimately user-controlled, RabbitMQ does not open channels, your applications do. So it's your applications that open 12K connections. We cannot know why.

Limiting the number of connections allowed on a channel, monitoring connection churn, identifying connections that use an excessive number of connections and ultimately identifying what applications leak or excessively use connections is the only recommendation Team RabbitMQ can provide given a three sentence long problem description.

Making all connections your applications open use easy to identify names will help.