> Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload.
It's not the size of the queue supported by the queue service that's the problem, it is the size of the queue in relation to the receivers capacity to process it. The infinite buffer can become a liability when your receivers are unable to handle it.
> 1) if your messages are important, suck it up and add more subscriber processing capacity or
>
> 2) if they're not important, purge the queue and restart your subscribers.
But what if things aren't so binary? Say I have a queue of emails that have backed up because my email service (or one of its dependencies) had a multi hour outage on black friday. Some of these messages are time sensitive ("We're packing it!" for a delivery that already happened) and some must be sent ("Your account is being terminated, this is the last notification"). You would like to discard messages like the former (which are certainly the overwhelming bulk of your messages) while making sure that you absolutely send the latter. Oh, and it turns out your email service has a fixed rate limit on your account and they won't raise it. This stuff happens.
Another example I've seen is a queue used as a workflow system: I enqueue commands (add X to Y, ensure X is in state Z, remove W from Y) in a backing system because, right, sometimes the dependencies for doing these operations are down for a little bit. But then one day they are down for a long-time, and the automation that has been set up on top of the external API has enqueued A LOT of commands. And a lot of those commands are now useless, things like an unprocessed "add X to Y" when there's a "remove X from Y" just a couple thousand messages later. My dependency is working to come back up to full capacity and isn't going to give me a higher rate limit, so I need to figure out how to drain this queue faster to get back to my normal line rate.
Another way to put this is that queues introduce bimodal state. When the queue is empty or only experiences temporary periods of overfill, my service runs in one way and has predictable behavior. But when my queue becomes chronically overfilled, my service is in an entirely different state. This is certainly bad news for my system, but can also be bad news for adjacent parts of the greater system that expect certain behaviors out of my service.
you’d have to build that into your system, or build a synchronous system in front of you queue to provide these semantics. and does critical imply time sensitivity, necessity of delivery, or both?
It's not the size of the queue supported by the queue service that's the problem, it is the size of the queue in relation to the receivers capacity to process it. The infinite buffer can become a liability when your receivers are unable to handle it.
> 1) if your messages are important, suck it up and add more subscriber processing capacity or > > 2) if they're not important, purge the queue and restart your subscribers.
But what if things aren't so binary? Say I have a queue of emails that have backed up because my email service (or one of its dependencies) had a multi hour outage on black friday. Some of these messages are time sensitive ("We're packing it!" for a delivery that already happened) and some must be sent ("Your account is being terminated, this is the last notification"). You would like to discard messages like the former (which are certainly the overwhelming bulk of your messages) while making sure that you absolutely send the latter. Oh, and it turns out your email service has a fixed rate limit on your account and they won't raise it. This stuff happens.
Another example I've seen is a queue used as a workflow system: I enqueue commands (add X to Y, ensure X is in state Z, remove W from Y) in a backing system because, right, sometimes the dependencies for doing these operations are down for a little bit. But then one day they are down for a long-time, and the automation that has been set up on top of the external API has enqueued A LOT of commands. And a lot of those commands are now useless, things like an unprocessed "add X to Y" when there's a "remove X from Y" just a couple thousand messages later. My dependency is working to come back up to full capacity and isn't going to give me a higher rate limit, so I need to figure out how to drain this queue faster to get back to my normal line rate.
Another way to put this is that queues introduce bimodal state. When the queue is empty or only experiences temporary periods of overfill, my service runs in one way and has predictable behavior. But when my queue becomes chronically overfilled, my service is in an entirely different state. This is certainly bad news for my system, but can also be bad news for adjacent parts of the greater system that expect certain behaviors out of my service.