The most important part of this article is the concept of back pressure and being able to detect it. It's common in a ton of other engineering disciplines but especially important when designing fault tolerant or load balancing systems at scale.
Basically it is just some type of feedback so that you don't overload subsystems. One of the most common failure modes I see in load balanced systems is when one box goes down the others try to compensate for the additional load. But there is nothing that tells the system overall "hey there is less capacity now because we lost a box". So you overwhelm all the other boxes and then you get this crazy cascade of failures.
This is a good article about overload and back pressure. It also lists some tools in Erlang to solve these sorts of issues. It also mentions genstage (very) briefly.
Or just the ability to tell your clients to back off.
Hopefully your clients retry server errors with exponential backoff, if you lose a datacenter you can send half of the requests 503 until you're back at a manageable load.
Hopefully detecting the load/generating the 503 is really cheap.
System capacity isn't always limited by resources within your control e.g. seats going on sale for an enormously popular concert, you're immediately throttled down to the sales abandonment rate. Everyone else needs to enter a queue.
More broadly, it's super easy to find yourself in a situation where throughout is constrained by an external resource. Especially common in any business that has partners (i.e. most of us) or has some kind of fulfilment to do.
Well. The concert reseller still has to serve all the clients.
It may display that there are "no more tickets for this concert [at the moment]" but it shouldn't display a 503 error to half of the users and it shouldn't overload the external-ticket-reservation-system either.
That makes an interesting problem where some systems will have to handle queueing/buffering/waiting/erroring nicely for each other =)
> Running at a 2N (or more) configuration 24/7 is a great way to light money on fire.
Till DynDns goes down for 8 hours and you're the only player left in town.
Then you profit the whole day and afterwards you check the revenues you've not lost + the new users and revenues that came in with the circumstances + the support calls and the engineering emergency that were avoided... turns out all of that cover the yearly datacenter expenses by tenfold 8)
Let's say that one box is down (don't care why, boxes die all the time).
You have 3 boxes, they're all running at 100%, or worse if the load is not perfectly balanced (it never is). You've got bad latency and you're up for cascading failures. Your system is in bad shape.
Now, supposing you have to deploy your application. Gotta take a server offline temporarily, the load will be 100% on the last 2 servers left... that can only take 66%. Booya! You're down :p
That's what happen when people are cheap on servers, in high performance systems.
Yes, but his point is still valid: You can to an extent reduce the overprovisioning you need by adding more, smaller units.
In your example, two boxes would have meant downtime no matter the capacity of each box too. All your example demonstrates is a reason why you might want to have capacity that allows taking more than one box offline....
Which could be handled with bigger servers, or more smaller servers.
I write this to illustrate that you must have a significant batch of servers (about 5) in order to achieve costs savings (by using smaller servers) WHILE not affecting reliability.
Well, that is, if you care about rare cascading failures and performances issues.
Backpressure is more common than water across engineering disciplines and it isn't an integrated part of every distributed system out there? Isn't that a bit of an oversight?
It's also part of http in a lot of ways. Browsers can make N simultaneous connections to a server. If those requests get queued up on the server side, then the client won't proceed with a new request until one finishes.
The pattern I see all to frequently is asynchronous worker queues. Those can very easily undermine backpressure. The rise of Ruby/Python and their limited concurrency models has placed a taboo on synchronous operations. However, synchronicity natural lends itself to backpressure.
Hate to be a party pooper, but I'd like to give people here a more generic mental tool to solve this problem.
Ignoring Elixir and Erlang - when you discover you have a backpressure problem, that is - any kind of throttling - connections or req/sec, you need to immediately tell yourself "I need a queue", and more importantly "I need a queue that has a prefetch capabilities". Don't try to build this. Use something that's already solid.
I've solved this problems 3 years ago, having 5M msg/minute pushed _reliably_ without loss of messages, and each of these messages were checked against a couple rules for assertion per user (to not bombard users with messages, when is the best time to push to a a user, etc.), so this adds complexity. Later approved messages were bundled into groups of a 1000, and passed on to GCM HTTP (today, Firebase/FCM).
I've used Java and Storm and RabbitMQ to build a scalable, dynamic, streaming cluster of workers.
You can also do this with Kafka but it'll be less transactional.
After tackling this problem a couple times, I'm completely convinced Discord's solution is suboptimal. Sorry guys, I love what you do, and this article is a good nudge for Elixir.
On the second time I've solved this, I've used XMPP. I knew there were risks, because essentially I'm moving from a stateless protocol to a stateful protocol. Eventually, it wasn't worth the effort and I kept using the old system.
I think you misunderstand the problem we are solving here. We are not trying to solve this because our system can't handle it. We are protecting it from when Firebase decides to slowdown in a way that causes data to backup and OOM the system. Since these are push notifications that have a time bound on usefulness we don't care about dumping to an external persisted queue like RabbitMQ or Kafka (we rather deliver newer notifications faster, than wait for the backed up buffer to flush). Firebase also only allows 1000 concurrent connections per senderId with 100 inflight pushes (that have not received an ack) which means that only 100,000 can be inflight. Ultimately if a remote service is providing backpressure because it is having a struggle no amount of auto scaling on your end is going to help you.
This service buffers potential pushes for all users being messages, that then watches the presence system to determine if they are on their desktop or mobile (this is millions of presence watchers and 10s of millions of buffered messages), and users are constantly clearing these buffers by reading on the clients and finally when a user is offline or goes offline we emit their pushes to them (which is what this article talks about). This service was evolved from our push system from the game we worked on and when it just did pushes only and no other logic it could push at 1m/sec in batches, but its responsibility has changed.
Context definitely matters - thanks for the background info. I understood what you're trying to do, incidentally I did the same (all these details sound familiar to me). At the time, Storm helped me batch, break batches and validate discrete units, re-batch, aggregate, repeat that process how many times I wanted, and finally, batch the stream with a strategy I wanted (number of users, messages, or balance number of connections) and deliver to Google.
Then I would just say, XMPP is new and fancy, but consider the old fashioned stateless HTTP interface. When I was implementing my own service, I was worried Google is not going to handle the load. Since we were partners with Google for a good while I was able to climb the ladder of people to get an answer, and plow through their closed-door policy for questions such as "Will you guys handle this load? (5M msg/min)". I wrote a huge email explaining every edge case and what I'm doing. The answer was "We will handle it.". No detail, no context, no buts. I wasn't confident at all. But in the end, they did handle it :)
Could you not reach pretty much the same result with a queue, though?
For example, workers could discard messages older than some threshold, quickly emptying the queue if there are expired messages. Clients might not even queue messages if the queue is currently too long, perhaps even providing a convenient signal for them to back off from their most chatty behaviour.
Some messages will not be delivered on time if there is significant backpressure. There is not much you can do about it, apart from avoiding choking yourself.
Perhaps the queue could work with a LIFO policy, to help at least some messages go through in time instead of having most messaged delayed near to the expiration threshold.
An all encompassing answer would be - the abstractions. Why would you use an operating system if really you have your CPU documented and know all of the instruction sets?
I've no idea about kafka but rabbit offers you message ack/nack and publisher confirms. Generally, you can build very solid things on top of it, depending on whether you need to distribute rabbit or not.
Anyone have any nice links to describe more about queue prefetching as described in this case? My google skills are failing because of all the CPU related articles.
This worries me a bit. At the moment they are providing the software and hosting it all absolutely free.
I would happily still use Discord if they provided the exact same thing with a monthly fee. Hopefully at some point they throw in some extra features for a pro version and start charging.
I just use Discord for gaming and haven't used Slack a lot, but I think Discord will be great for work as soon as they release search.
I don't know. I use Slack and Discord a lot. The only advantage Discord has going for it is the voice chat channels, which has much more utility for gaming than business in my opinion. Otherwise Slack is way better at all the other features and I usually feel neutered when I'm using Discord after using Slack all day.
If they need ideas for a pro version I'd probably pay far too much to be able to record each user into individual audio files (recorded locally to each user and combined on my system) for podcasts, letsplays (YouTube videos), remote meetings, etc
This is doable today. Streams are mixed client-side. The downsides are compression (Opus at 96kbps is pretty good, though) and loss of audio mixing (echo cancellation, noise suppression) secret sauce, though this could definitely be reapplied. If this is worth money to you even with these downsides, hit me up.
> Discord is always completely free to use with no gotchas. This means you can make as many servers as you want with no slot limitations.
Wondering how we’ll make money? In the future there will be optional cosmetics like themes, sticker packs, and sound packs available for purchase. We’ll never charge for Discord’s core functionality. [0]
Something smells very fishy there. I have a hard time believing they got a $30 million investment by pitching stickers, add-on themes, and sound effects...
Isn't that the core functionality that they say they won't charge for?
If not, I don't understand why they bothered mentioning "how will we make money?" on their website, and then why they didn't answer with, "By charging game vendors a ton of money for the SDK!"
What's the point? Tons of game engines already offer that in-engine, eliminating the need for Discord's ability to do that. (It's been around since UT2K3 earliest I can remember.)
No, I haven't tried steam VoIP because as mentioned, the game engine itself usually comes with a working and tested solution, so why bother using any other VoIP application that's just wasting RAM and CPU cycles trying to do the same thing in a far larger footprint?
In the case of Overwatch you can group up with friends outside the game and share your own voice channel with each other, but when you join a competitive match you have to choose between group chat and team chat, and most people expect you to join team chat.
In other words, if you and one friend join a game you can either talk/listen to that one friend alone or the entire 6 man team, and switching between the two requires multiple menu clicks.
So we use Teamspeak alongside the game to allow us to talk privately with each other seprate from the whole team in game. With any modern gaming PC the "wasted RAM and CPU cycles" are negligible.
Stickers are more than something to sell to consumers - They can be advertisements used to target a gaming audience. Skype and Facebook have done this, offering free sticker packs for upcoming movies (e.g. Inside Out). A pack of stickers or a theme featuring characters from a game like Overwatch could potentially bring in revenue from both Blizzard and players.
They have an awful lot of information about video gamers in conversation history. They could mine that data for game companies and sell it as a way to help companies build better, more addictive and mechanically pleasing games.
I'm not sure how useful that is. It's not like gamers' opinion about games are hard to come by. Gamers are very vocal about their opinions, so any game developer looking for feedback can just go to Steam/Reddit/NeoGAF/whatever and read to their heart's content, or maybe even communicate directly to their player base.
What people say/believe they like isn't necessarily consistent with their behavior. Data-mining with sentiment analysis might be useful. ("'Wows' dropped 1.3% and negative-sentiment profanity rose 0.5% after the latest patch.")
Isn't it a little irresponsible to make that claim while operating on VC funding with no monetization strategy?
I don't mean to imply that you yourself are being dishonest or that you will go back on your word at some point. I'm sure you have noble intentions and want the best for your users. But it might be beyond your control.
If the company gets sold or goes public, you could have drastic changes in management and their philosophy. You acknowledge this outright in your privacy page, under the "OUR DISCLOSURE OF YOUR INFORMATION" heading:
Discord has a policy in place that explicitly says they can do whatever they want with the information without limit, including selling it or transferring it without consent. This directly conflicts with employees' public claims about intentions.
The difference is that one of them is a legal acknowledgement end users must make before they use the software and the other is a 'feel good' thing to hear on someone's blog or forum post.
If they just wanted protection against an accidental slippage of data then that privacy page could be changed substantially. Instead, they pave the road for the explicit sale of data at a later time.
Or, if they wanted to leave the option open in the future they could say "This privacy policy is subject to change" and give users an opportunity to opt-out when it changes without historical data being subject to undisclosed future use. But this weakens their value in an eventual acquisition by someone who wants to monetize the data.
As it stands right now, in an eventual acquisition or even just some internal shifts of philosophy in the organization, all historical data is up for grabs for any potential use.
Discord really shouldn't have employees state that they will never sell users data when they explicitly allow and plan for that option. It may not be intentional dishonesty, but it comes close.
"The difference is that one of them is a legal acknowledgement end users must make before they use the software and the other is a 'feel good' thing to hear on someone's blog or forum post."
No, if it's from a known employee, it counts as a legal advertisement.
Tencent demands way too much for what they pay. Even when it comes to consultants and our already-exorbitant pricing. They may not say it immediately, but it is coming.
Oh, fuck no. No, no, no. I can not think of a more abusive thing for a company to do to its customers than that suggestion right there. How little respect would a company have for their fellow humans that anyone could even consider such a move?
Maybe that's just a failure of imagination on my part, but I'm ok with that.
Aggregated or Non-identifiable Data: We may also share aggregated or non-personally identifiable information with our partners or others for business purposes.
It is, but slack seemed to go for a more corporate angle (along with Microsoft Teams), and there's not really anything for more casual users. On the other end of the scale we have Skype but that's more for personal chatting and their group conversation support is very week.
They have found a pretty good niche with a lot of users. Just like how reddit did with subreddits. There are a lot of "smallish" communities out there, based around games, likes, twitch/youtube channels, etc, and they all need some place to hang out when they're not in saig game/channel.
I think right now what matters most is who comes out winner with the biggest community, monitezation can come later.
It is bizzare to me that Google's hangouts product didn't end up filling this niche. I cannot understand why they followed FB lead instead of working to dismantle the attraction of closed messaging systems. Topical group chat seems like it would just naturally fall out of circles, hangouts, and wave.
I sure hope one possible end-game is to re-skin it without the gaming focus and price it competitively to Hipchat/Slack.
As soon as screensharing lands it'll be an across the board upgrade to Hipchat at my work (only missing feature I can think of is video calls, but quality over hipchat has always been a bit sketchy so we usually fall back to Hangouts).
They're in the gaming market, there's cash plenty. Even if they just went the advertisement route, which I doubt they will they should end up doing well.
That's awesome and it just goes to show how simple something can be that would otherwise involve a certain degree of concurrent (and distributed) programming.
"Obviously a few notifications were dropped. If a few notifications weren’t dropped, the system may never have recovered, or the Push Collector might have fallen over."
How many is a few? It looks like the buffer reaches about 50k, does a few mean literally in the single digits or 100s?
Good question. We don't have metrics on the exact number dropped. We're using an earlier version of GenStage that doesn't give any information about dropped events. Once we upgrade we'll have a better idea.
That seems too important to have zero visibility on to me. Just eyeing the graphs, your queue size grew at 750m/s from 17:49 to 17:50. You then starting shedding at 17:50 for 40s. Assuming the ingress rate was roughly linear (which it looks like it was) you shed ~30,000 requests out of 3-4M. Does that not seem high to you?
This system seems great for at most once delivery. I wish I had more problems to solve with that constraint.
Yep, such is life with new tech. GenStage was 0.3.0 when we started using it. We'll be able to get this visibility once we update to a newer version (hasn't been prioritized).
FWIW, the buffer only fills up to the peak about once a month. Load shedding is the last ditch effort in catastrophic situations so that everything doesn't fall apart.
Curious why you don't have metrics on the other end (whatever is sending to the "Push Collector")?
What if the Push Collector is down or has a random bug where it throws away XX% of requests for no good reason? How would you know if you don't instrument the other end? Something like StatsD works fantastic for this, but also just logging those failures and using a log search/aggregation tool like Kibana or Splunk would be a step in the right direction.
There's another important question: How will the clients deal with the fact that they did not get a notification delivered? Will that mean they probably never receive a chat message? That could in some cases be catastrophic for the user. Or would it only mean that they may not get something instantly, which would not be too bad if the client would also poll the server or also try to catch up on notifications on reconnects.
When the push notifications hit FCM, Firebase do not guarantee delivery of those messages to clients (usually iOS or android devices). There are quite a few reasons that FCM/APNS might fail to deliver a message, so applications almost never have functionality depend on them.
As you say, you might not get the notification pushed to the device, but you should still see the message if you open the messaging app as normal.
This is indeed the case. Our real time system is outside of firebase and APNS and it handles the actual real time updates of chat state once the app is launched. We also have a delivery system that accounts for network cuts/switches and the like.
If the buffers are filling faster than the servers can clear them, then you're headed for "catastrophic" failure anyway and notifications are going to get dropped regardless. You can handle it more or less gracefully while you spin up more capacity.
The other choice is to always keep around more spare capacity, but that can get expensive if as they describe these drastic peaks happen once a month.
I like to differentiate into 2 categories of "catastrophic": The one you mention is that the service can't keep up with the demand and that is has to take some actions to stay alive and not crash. That's and important thing which should be incorporated into the design.
The other category which I meant is that in such situations the system should not run into inconsistent/weird behavior which is catastrophic from the end user point of view (not the system). E.g. in a chat application if user A sends a message to B and on his client he gets an acknowledgement that the message has been delivered. However if the server under high load simply drops the message before forwarding it to B that user might never get it. A sees that something was delivered while in reality it was not. If A depends on that information it surely might be catastrophic to him.
All in all you should have a complete system design that even under high pressure works deterministically for the users. E.g. user A only gets an ACK that the message was sent after it was somehow persisted on the server. And if the server can't deliver the message to the other client because it was dropped it is still marked somewhere for retry later on. Or it will get fetched at a later time by the client through some poll operation.
- To avoid OOMing the Erlang VM.
- If the notification queue is backed up then older notifications are not worth delivering if we can speed up delivering of more recent ones.
Except that's not what's happening here if I'm reading right. They are throwing away the newer ones and not the older ones because the ones rejected by the Push Collector are the most recent ones.
I was wondering the same thing. Dropping an unknown number of requests isn't all that impressive. It seems like a simpler approach would have been to use a Message Queue of some sort with pushers pulling items from the queue.
Not sure why are you getting downvoted, I came here to make the same comment.
QPM is a useless metric. When talking about distributed systems from engineering point of view, you always want to use QPS. QPM is simply not fined-grained enough to show whether the traffic is bursty or not. For example in this particular case, when you say 1M QPM that can mean anything - they might be idle for 50s and then get 100k QPS for the next ten seconds, or they might be getting 15k QPS all the time (like it's visible on the graph). Distributed systems are designed for the peak workload, not for the average one. Using misleading numbers like QPM leads to bad design and sizing decisions.
The only case where you would use QPM, QPD and similar metrics is when you want to artificially show your numbers bigger than they are (10M transactions a day sounds better than 115 transactions a second). But those should be used by sales, not by engineers.
I read it originally as 1M QPS, and thought that was a nice number. It was upon further inspection that I saw it was 1M QPM, and I was no longer intrigued.
I'm not sure if my org qualifies. Depends on how you count it I guess. We have 80k+ RPS at times at SendGrid, and each request can generate 4 to 8 external events and at least a dozen internal API calls. If you count total internal QPS, that would be something in the order of 80k * 4 * 12 ~= 3.8M QPS. I'd have to check with an operations person to see if that checks out. I don't know if it is fair to count this though. So, let's go back to the 80k RPS. If someone was doing 10X that, I'd be intrigued to learn more about their set up for sure. I imagine the Googles, Facebooks, and Amazons of our industry do this level of traffic.
Here's a cool trick I figured out. If you have something measured in units per minute, you can divide it by 60 to get units per second. I won't even charge you to use the method even though I'm in the process of patenting it.
The requests per second number should be given for peak load. That is a very important metric, a system has to be scaled to sustain the peaks, not the average.
We'd need to know the traffic pattern to know the multiplier, that is certainly not 60 :p
50k seems like a low bar to start losing messages at. If this was done with Celery and a decently sized RabbitMQ box, I would expect it to get into the millions before problems started happening.
These machines do more than just push. They also buffer messages for each individual user to "potentially" push if they don't read them on the desktop client. This happens before the flow this article talks about.
We currently have 3 machines doing this for millions of concurrent users. At the writing of this article it was 2 machines.
What size machines are these? I'm shocked that this volume is your max handling with Erlang unless your using a smaller T series AWS instance for this.
These are getting easily over 30,000 requests a second each about updating queues for new messages. And also are subscribed to presence events from our presence system to millions of people. It is a very busy service ensuring we only deliver messages to people not at their computer.
So if they are delivering 30k a second per box and the max "backlog" you allow to build is ~50k, then you cap your backlog at under 2sec worth of delivery? Or am I missing something?
At some point, when a system has entered a failure mode for a while, it makes sense to start shedding load, rather than attempting to deliver every single push notification. Also worth mentioning, a minute of downtime is already a million backed up pushes. Beyond that, it becomes infeasible to attempt deliver them.
Edit: Also worth mentioning, the 50k buffer is for a single server, we run multiple push servers in the cluster.
At 15k notifications per minute, a million notifications would take 1hr to clear before the queue returns to normal. I would imagine they prefer to shed load early so notifications don't get delayed, hence the small buffer.
The issue was not the ability of their servers to handle the load, but the ability of Firebase to ingest the notifications - at least, that's how I read it.
I love Discord, and love Elixir too, so this is a pretty great post.
Unfortunate that the final bottleneck was an upstream provider, though it's good that they documented rate limits. I feel like my last attempt to find documented rate limits for GCM/APNS was fruitless, perhaps Firebase messaging has improved that?
It does, doesn't it? I used to use Ventrillo, but then they screwed our small group out of our server connection. And we happily used Dolby Axon for a while. We tried Google Hangouts... for a while; until it just really didn't work well (it just disconnected and crapped out a lot). We tried using the Steam client's chat, but while ok for screensharing, it wasn't so great for chat.
But at some point we heard of Discord, which posed itself as a chat/vent replacement, started using it, and it just works. Which is huge, since the other stuff generally didn't (Axon was actually good).
Ventrilo is laggy compared to TS and Mumble. It won't show the lag as ms, but it is there, and it is real. Its due to the way the protocol works, or it is the server. It is no longer in development, and you can't even run your own server on your own hardware. The interface is from the 90s. You don't want to use Ventrilo for gaming in 2016.
TS supports plugins. No lag issues, can run on your own server. Closed source.
Mumble is open source, no lag issues. Interface is slightly less good than TS. Supports SSL.
Discord, like you say, Just Works (tm). It is very easy to use, the interface is amazing, its in active development, and setting up a server is free. It also works in the web browser.
If you're into Blizzard games, Battle.net recently added native VoIP in their client. The advantage that has as Blizzard gamer is you don't have to install any 3rd party software.
This is what makes Discord so good. Before Discord, when I wanted to play online with a friend for the time, I'd have to convince them to download Mumble or Ventrilo, teach them how to connect to a server, and help them set up their mic. With Discord I just send them a link and we're talking. They can get the client later, and having a persistent chat area is a fun way to build up a sense of community.
They had a really well defined user-space, marketed at it well, and really nailed the user experience, while still being free for the typical user. There is a lot to love about Discord.
I don't have anything particularly negative to say about the competition, but if you're looking for a completely free and open infrastructure to build on, I don't think anything (besides IRC, but IRC has it's own problems) comes close to Matrix at the moment. I'm very impressed with the project so far. That said, with a reference server implementation in Python, and few native desktop applications available, there is plenty of work to do.
Edit: Matrix already works great as an IRC bouncer for networks which have been bridged.
IRC is a pretty not-extensible platform. It's just not in the spec to have larger messages, metadata. Then you get into things like push notifications, image hosting, video hosting, file hosting...
IRC for the 21st century IS these apps like slack, discord, etc.
IRC is for people who participate in the internet. For them it isn't hard to host their own website or just use any of the innumberable image/video/whatever hosts and paste a link. Discord is for tech ignorant people who consume things pushed to them by companies over the internet.
Bundling up everything into a centralized, proprietary package will go bad for the consumers eventually. But for now they'll trade away their freedom for convenience and enjoy it.
I don't see the justification for speaking poorly of their users. Is convenience a bad thing now? Why do I want other image hosts hosting my images instead? What's the difference between putting them there and a chat platform from a privacy or centralization standpoint? I know for a fact many people use slack to trade images they don't want on the public net. So then I need to host a private image host too? That's a bit silly.
IRC is a fine platform. That doesn't mean it's what everybody needs/wants.
Discord has a compact layout option. It's good enough for me--the pixels wasted here and there improve readability while not losing too much text IMHO.
Then again, using a monospace font like I did for years (ssh+screen+irssi) wastes a lot of screen area just because every character is so wide.
I ended up there for the first time last night and must say that there is a lot to like about it. I found some good communities and integrating media and so on all felt quite streamlined, and the system was snappy.
It's just too bad there are a dozen IMs/voice/video and a dozen slacky/feed companies.
There was an initiative not long ago called Reactive Streams which established some common interfaces to build things like this. Back pressure was one of the main concerns.
I spend a lot of time in the PCMR Discord, which is pretty lively. The technology seems to be solid, while the UI has issues (notifications from half a day ago are really hard to find for example on mobile devices). Otherwise I'm on Discord every day and love using the service. I miss some slack features, but the VOIP is very good.
What features in particular? The most common one we hear is search, which is actually implemented and undergoing internal testing before a public preview soon.
It's just what I mentioned. I'll get a notification, and I just can't find where I was notified from. Like on Android, if I click on the notification I would expect it to take me to where the conversation happened where I was notified. It would take a really long time of scrolling to try and find the notification given the volume of discussion that happens. Can I just like click on something to see all my notifications from android, click on them and go to the conversation?
16k per second. 83k per second during peak (assuming 80/20 default traffic rule).
- 100 /s = typical limit of a standard web application (python/ruby), per core
- 1.000 /s = typical limit of an application running on a full system
- 10.000 /s = typical limit for fast systems (load balancers, DB, haproxy, redis, tomcat...).
- Over 10.000/s You gotta scale horizontally because a single box [shouldn't] can't take it.
The difficulty depends on the architecture and what the application has to do (dunno, didn't go through the article). You make something that can scale by just adding more boxes, then it's trivial, just add more boxes. Well, it's gonna costs money and that's about it.
So no. Not a big deal at all... if you've done that before and you've got the experience :D
While 1k/sec seems to be an average throughput for most web apps due to all the logic, 10k/sec is nowhere near the limit for fast systems, many can do well into 6 figures per second with some now doing millions/sec.
Your description for fast systems refers to high-performance software like load balancers and databases. In this case, 10k is nowhere near the limit on modern machines, they all do 6 figures per second.
With good hardware and good tuning, possibly. Again I'm not saying it's a hard limit, I'm saying it's a reasonable expectation for real world production usage ;)
At 100k requests/s on a load balancer. The HTTP headers alone (450 bytes) with zero content are more than 1300 MBps of traffic. And well, the CPU will bottleneck long before that amount of requests to parse.
Everything is relative. In this case, it's not so much the actual load itself but rather the throttling ability to match the upstream provider's throughput and limitations.
Not exactly. Without running on the BEAM you're left with cooperative scheduling (handing back control to the scheduler) of processes instead of pre-scheduling (the scheduler can stop you).
That makes it possible for one processor heavy operation to take over and slow down everything else. BEAM ensures that if you have millions of request coming through and suddenly 1000 4 day long operations kick off on the machine, that the millions of normal, smaller operations continue responding and performing as expected.
Fairly critical for the stability of real time systems.
The other piece here is that these processes are cheaper on the BEAM than any other platform in terms of RAM cost.
.5Kb / process on the Erlang VM. A goroutine in Golang is the next closest at 2kb.
The two combined are one of the big reasons why benchmarks don't tell the whole story with Erlang/Elixir. It's harder to measure consistency in the face of bad actors.
It is constant - but iirc, it'd be trivial to make a dynamically scaling pool. At the end of the day, a pusher is just a TCP connection. Keeping a pool of fixed size and planning capacity around scaling horizontally is a perfectly acceptable approach - given you know the potential throughput for each pusher.
just wondering, what is the difference if I use two kind of [producer, consumer] message queues (say rabbitmq) instead of this? Does genstage being a erlang system makes a difference?
RabbitMQ is written in erlang. So basically you use it natively instead of bringing and configurating a big dependency. It just come with your language for free without needing another process, etc etc.
how does one achieve this in Celery 4? I remember there was a celery "batch" contrib module that allowed this kind of a batching behavior. But i dont see that in 4
> "Firebase requires that each XMPP connection has no more than 100 pending requests at a time. If you have 100 requests in flight, you must wait for Firebase to acknowledge a request before sending another."
So... get 100 firebase accounts and blast them in parallel.
Basically it is just some type of feedback so that you don't overload subsystems. One of the most common failure modes I see in load balanced systems is when one box goes down the others try to compensate for the additional load. But there is nothing that tells the system overall "hey there is less capacity now because we lost a box". So you overwhelm all the other boxes and then you get this crazy cascade of failures.