AWS has sqs & kinesis which are much better queueing options in that scenario where you’re in AWS doing new dev and can pick a technology. This is more likely to do with opening doors for large and complex applications that can’t be rewritten to come into the AWS cloud. Unsexy stuff but there’s some fun engineering to be done in that realm, if you find like puzzles and shitshows to be fun anyways :-)
how are those much better? if you use SQS and you write your code for it, then you are stuck on a proprietary platform. Also, SQS is super-basic and actually requires a bunch of code to do anything beyond trivial - although yes, it seems reliable and well-supported, at least from my experience. I was actually really waiting for AWS to support Rabbit since it seems to hit the right combo of features, usability and platform independence for me, and it looked friendlier than ActiveMQ.
If you're using a decent framework there's a good chance it already does most of the work for you. With Ruby on Rails, there's the ActiveJob abstraction which you can hook up to different backends like SQS or Redis with a few lines of code. In addition, AWS has a lot of out of the box integration with SNS and SQS for other services like Cloudwatch and Lambda (in fact, lambda you don't need any special code).
If you have a Lambda function processing SQS messages they just get dumped in your handler method and it your function runs successfully they get automatically removed from the q. If your lambda fails, the message reappears after the visibility timeout out subject to your redrive policy
I'm referring to things like topics, multiple consumers, routing, etc etc that are not even possible with SQS and once you grow into a need for those, SQS stops being adequate no matter what library you use for it.
They aren't. They're technically more correct but not always the practical best choice.
RabbitMQ is a smart play as Rabbit is very easy to use, understand, and troubleshoot at the low end (which is where I suspect the vast majority of queue systems live).
It also has a feature which is actually really hard to do (and sqs doesn't do). Guaranteed delivery of a message once.
That was THE reason we never migrated to SQS, there are scenarios where SQS can double deliver. Our codebase was built up from nothing over time and couldn't gracefully handle double delivery of messages in all scenarios. We could have refactored, but it wasn't worth the work when we were already doing a half billion in revenue without getting even close to the limitations of rabbit AND were close to selling (which we ultimately did).
AWS is great at selling multiple slight variations of the same product. If you look you can usually find ONE variation that works for you. The real test will be if the billing isn't garbage (garbage billing is why we didn't use their other AMQP service and part of the reason why we don't use things like EKS or Managed SFTP despite having the need).
That flies in the face of my distributed systems knowledge. It's not possible in some failure cases.
If your acknowledgement of a message gets lost (because either server involved or the pipes in-between fail) you've processed the message already but the queue server will think you haven't. It either has to resend it (duplicate delivery) or it ignores acknowledgements all together (drops messages that it sent you, but you didn't process - maybe because your server failed.) So the choice when there is a failure in the system is between at least once or at most once - exactly once cannot be guaranteed.
You are correct, a better description is that their path to 'deliver exactly once to the best of your ability' is clearer.
If I remember correctly SQS is hard limited to a fairly short timeout to requeue messages delivered but not acked. In rabbit it's much more configurable.
Also regular rabbit hosts support the kludge pattern of, 'just run one host and accept if it goes poof you can lose messages,' which is useful if you don't want to bother with the complexity of clustering or are on a shoe string budget.
Lastly you get a nice user interface with the management plugin and you can stand it up locally with docker compose (without depending on AWS for dev or any of the 'aws but on your laptop' solutions).
They’ve been around for quite some time so they have a much wider customer base & bigger teams supporting them. AWS services can be a bit choppy in the beginning so imo, especially with queues, I’d wait for it to bake.
