"Finally, GM is not at all designed to cope with partial network partitions: if A is partitioned from B, then B can remove it from Mnesia, and that information can leak back to A via C. We currently don't handle this situation well; this is the biggest unsolved problem in RabbitMQ."
Thankyou for acknowledging this! The mirroring is functionally not a very useful HA/horizontal scale feature until this is sorted. Mnesia has the same problem, so one has to imagine that the root cause is Mnesia.
Running RabbitMQ in production for large scale OpenStack deployments, I have actually started to recommend a singleton RabbitMQ instance because the mirroring can cause all kinds of emergent behaviour during a network partition.
OTOH, RabbitMQ does handle OpenStack at scale very well when there are no partitions, which is no mean feat, so kudos and thankyou.
Indeed! We stopped using RabbitMQ for this reason. Actually, for a number of reasons (its slowness being another major one), but this was the biggy. As a fault tolerant system it doesn't tolerate faults very well at all; we just could not find a way of keeping it stable for long periods, so it had to go.
Thankyou for acknowledging this! The mirroring is functionally not a very useful HA/horizontal scale feature until this is sorted. Mnesia has the same problem, so one has to imagine that the root cause is Mnesia.
Running RabbitMQ in production for large scale OpenStack deployments, I have actually started to recommend a singleton RabbitMQ instance because the mirroring can cause all kinds of emergent behaviour during a network partition.
OTOH, RabbitMQ does handle OpenStack at scale very well when there are no partitions, which is no mean feat, so kudos and thankyou.