[Suggestion] CMR for streams

[tschuettig]

RabbitMQ series

4.1.x

Operating system (distribution) used

Linux (Ubuntu)

How is RabbitMQ deployed?

Community Docker image

What would you like to suggest for a future version of RabbitMQ?

At the moment RabbitMQ supports Continuous Membership Reconciliation (CMR) for quorum queues, but it seems like there is no feature that achieves the same for queues of type 'stream' (except for using explicit replica management). Is it possible to introduce this for streams in the future or is there a reason why this should not or cannot be implemented?

[kjnilsson]

As stated in https://www.rabbitmq.com/docs/streams#member-management

member changes for streams isn't as safe to do as for e.g. quorum queues which means that we're probably not quite ready for any such automation.

[the-mikedavis]

@kjnilsson let's chat about this next time we sync up. We'd be interested in improving the stream coordinator and eventually contributing CMR when we think the stream coordinator is ready for it

[SimonUnge]

@tschuettig Would love to hear more about your use case for it, and if you have any interest in helping with contributions/design of the feature. As @the-mikedavis states above, we are also interested in a feature that would automate stream membership in certain situations.

[tschuettig]

My use case is the recreation of RabbitMQ cluster nodes, e.g. when the RabbitMQ pod is moved to a different node inside Kubernetes. We would like to achieve this without transferring any persistent data across Kubernetes nodes which means the mnesia table will get lost. If the streams are not replicated, the stream queue would be deleted after every RabbitMQ node has been recreated. Our current approach is to replicate the queues inside the Kubernetes statefulset command instead.
Regarding the contribution, I barely have any experience with Erlang and probably also lack the time capacity.