Rebalancing fails partly when one of two clients with multiple single active consumers (with the same name) on a superstream crashes #13372
-
Community Support Policy
RabbitMQ version used4.0.5 Erlang version used27.2.x Operating system (distribution) usedLinux version 5.15.153.1-microsoft-standard-WSL2 How is RabbitMQ deployed?Community Docker image rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics Logs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs rabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location Steps to deploy RabbitMQ clusterdocker-compose up glrabbitmq: Steps to reproduce the behavior in questionA super stream 'CmmnEvents' is created with 8 partitions. advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application code# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKSWhat problem are you trying to solve?When multiple client applications define multiple single active consumers (with the same name) on the same partitioned superstream, and when one client crashes, we expect the remaining client to take over consuming all the partitions, but this scenario always ends with 2 of the 8 partitions unconsumed. |
Beta Was this translation helpful? Give feedback.
Answered by
acogoluegnes
Mar 28, 2025
Replies: 2 comments 5 replies
-
You deploy 2 instances of the same application, is that correct? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
yes that is correct |
Beta Was this translation helpful? Give feedback.
-
|
I could not reproduce the issue. Can you provide a step-by-step procedure? Here is the procedure I used. We expect something similar from your end to help us diagnose the issue. Start a 3-node cluster. Using the stream Java client Docker setup is convenient: (to stop the cluster later: In another terminal tab, get Stream PerfTest and run it to simulate a super stream consumer (it creates the super stream as well): In yet another terminal tab, run another consumer: List the consumers of the group for the List the Java processes with Kill one of the Stream PerfTest processes: List the consumers on the There is still a consumer, it was inactive before and has been promoted to active. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Hello, thank you for this setup suggestion. we were able to reproduce our issue with it: the issue arises when the clients define more than one consumer: java -jar stream-perf-test-latest.jar --producers 0 --consumers 2 \
--stream-count 1 --super-streams --super-stream-partitions 8 \
--single-active-consumer --consumer-names my-app \
--uris rabbitmq-stream://$(hostname):5552,rabbitmq-stream://$(hostname):5553,rabbitmq-stream://$(hostname):5554 \
--consumers-by-connection 100if each stream-perf-test defines two consumers (this also happens in our real client application), we can see that each of the 8 partitions lists 4 consumers whereof 1 is active, like this: But after 1 client is killed we see that each of the 8 partitions has 2 consumers left, but only 6 of them have an active one, and 2 partitions remain with 2 inactive consumers: So at this point we have 2 stream partitions without an active consumer. |
Beta Was this translation helpful? Give feedback.
-
|
@acogoluegnes, Did you have the chance to verify/reproduce this issue (with the same procedure with 2 Stream PerfTest clients but with 2 consumers for each client) ? Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Your comment went under the radar, sorry about this. I'll have a look. |
Beta Was this translation helpful? Give feedback.
-
|
I identified the problem, I'm working on a fix. |
Beta Was this translation helpful? Give feedback.
Answer selected by
michaelklishin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I identified the problem, I'm working on a fix.