Hacker News new | past | comments | ask | show | jobs | submit login

How does RabbitMQ compare with Kafka?



I'm more familiar with SQS than RabbitMQ, but have used both, and have chosen between queue and stream based solutions.

Kafka is a stream, and can be replayed (if you have it set up to store stuff). Rabbit is simply a queue, and when the messages are gone, they're gone.

This means that queues are a lot smaller, but can only serve one set of consumers at at time. If you want to have multiple things listening to messages, you have to use fan-out patterns that place messages on multiple queues. Queues can also suffer from less than atomic delivery, especially if the system is distributed. This means you have to jump through some hoops and add an atomic layer somewhere if you want to ensure you're not double processing anything.

Kafka can have infinite retention (if you got the storage/$), and you don't need to have multiple streams to service multiple consumers. Each consumer stores where they are in the stream, and can traverse as needed. You'll need to be careful to make sure that a single consumer is handling a single partition to promise that you'll only process a message once.

Managing streams can be a headache, but less so now if you have money to have Amazon or Confluent manage it for you. They offer pretty much unlimited scalability, and are the production grade solution for a ton of problems.

Queues are really simple to understand and build and still scale pretty dang well. Just make sure your message processing is idempotent and make sure you can handle if something is processed multiple times.


I've been interested in this question as well. There's a lot of sources online comparing the two but none really definitive.


RabbitMQ is not suitable for event sourcing. Kafka is. In general, RabbitMQ is a “river” and Kafka is a “lake”.

RabbitMQ has excellent support for complex message flow topologies. Kafka out of the box does not provide these features.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: