"Finally, GM is not at all designed to cope with partial network partitions: if A is partitioned from B, then B can remove it from Mnesia, and that information can leak back to A via C. We currently don't handle this situation well; this is the biggest unsolved problem in RabbitMQ."
Thankyou for acknowledging this! The mirroring is functionally not a very useful HA/horizontal scale feature until this is sorted. Mnesia has the same problem, so one has to imagine that the root cause is Mnesia.
Running RabbitMQ in production for large scale OpenStack deployments, I have actually started to recommend a singleton RabbitMQ instance because the mirroring can cause all kinds of emergent behaviour during a network partition.
OTOH, RabbitMQ does handle OpenStack at scale very well when there are no partitions, which is no mean feat, so kudos and thankyou.
Indeed! We stopped using RabbitMQ for this reason. Actually, for a number of reasons (its slowness being another major one), but this was the biggy. As a fault tolerant system it doesn't tolerate faults very well at all; we just could not find a way of keeping it stable for long periods, so it had to go.
I found RabbitMQ's docs to be pretty useful myself, as long as you are familiar with regular expressions. I set up a system with fewer users and simpler permissions though. Out of curiosity, what did you find lacking?
This will be helpful in understanding this framework I found yesterday called RestBus[1]. From their site: "RestBus is a high performance messaging library that enables you to consume ASP.NET Core, Web API and ServiceStack service endpoints via RabbitMQ".
Thankyou for acknowledging this! The mirroring is functionally not a very useful HA/horizontal scale feature until this is sorted. Mnesia has the same problem, so one has to imagine that the root cause is Mnesia.
Running RabbitMQ in production for large scale OpenStack deployments, I have actually started to recommend a singleton RabbitMQ instance because the mirroring can cause all kinds of emergent behaviour during a network partition.
OTOH, RabbitMQ does handle OpenStack at scale very well when there are no partitions, which is no mean feat, so kudos and thankyou.