Message queues are one of these things that sound great in theory but IME has simply been a way of adding more failure modes and points of failure to chase the (largely unjustified) goal of systems decoupling.
There are two types of people in the world: those who divide the world into two kinds of people and those that don't. As one of the former, I like to view MQs as being essentially one of two types:
1. Stateless
2. Stateful
Stateless means if there's no one there to handle the message, it is simply lost. The benefit of course is you don't need to consider all the complexity of persisting messages. But a stateless MQ is really just a routing / service discovery architecture. Now that's completely fine to abstract that away but it's not really that exciting. It's still synchronous delivery, essentially.
Stateful OTOH means you really can queue things up and deal with them later. Unfortunately you're now dealing with:
- Reliability: what if the persist call fails? Do you have enough storage? Can your persistence medium keep up with the volume of reads and writes?
- Is delivery guaranteed or is it simply best-effort? This is a big one. Best effort is MUCH less problematic but doesn't tend to be what people want. You see guaranteed delivery in a lot of enterprise MQ software (eg TIBCO certified messaging) where you have things like MQ integration with two-phase commits and the like.
3. What ordering guarantees are there? Guaranteed ordering or best-effort or no promise?
Basically you've probably just created another database. Worse, that database may be different to your other databases.
Also, is your MQ a FIFO or some form of priority queue? If it's a FIFO, how do you handle throttling and buffering?
The post describes the apocryphal MQ as having infinite capacity. No MQ is.
IME the supposed benefits have been outweighed by all the negatives.
> Basically you've probably just created another database
I think that's a fair assessment of persistent queues, using "database" in the most general sense.
> Worse, that database may be different to your other databases.
Different isn't inherently worse. It's just a different tool for a different job than a traditional RDBMS. The problem only occurs when we pretend they're the same.
There are two types of people in the world: those who divide the world into two kinds of people and those that don't. As one of the former, I like to view MQs as being essentially one of two types:
1. Stateless
2. Stateful
Stateless means if there's no one there to handle the message, it is simply lost. The benefit of course is you don't need to consider all the complexity of persisting messages. But a stateless MQ is really just a routing / service discovery architecture. Now that's completely fine to abstract that away but it's not really that exciting. It's still synchronous delivery, essentially.
Stateful OTOH means you really can queue things up and deal with them later. Unfortunately you're now dealing with:
- Reliability: what if the persist call fails? Do you have enough storage? Can your persistence medium keep up with the volume of reads and writes?
- Is delivery guaranteed or is it simply best-effort? This is a big one. Best effort is MUCH less problematic but doesn't tend to be what people want. You see guaranteed delivery in a lot of enterprise MQ software (eg TIBCO certified messaging) where you have things like MQ integration with two-phase commits and the like.
3. What ordering guarantees are there? Guaranteed ordering or best-effort or no promise?
Basically you've probably just created another database. Worse, that database may be different to your other databases.
Also, is your MQ a FIFO or some form of priority queue? If it's a FIFO, how do you handle throttling and buffering?
The post describes the apocryphal MQ as having infinite capacity. No MQ is.
IME the supposed benefits have been outweighed by all the negatives.