At that moment, we can't recover from that and players connected to one of the crashed exchanges (1 server process: N exchanges) are told to restart levels, which is not an ideal outcome but it's essentially free. I hope to eventually be able to do something like use our persisted-out-of-memory (e.g. to NSQ) tapes of the orders, bring up the crashed exchanges, and fire every order back into it in order, which (since it's a stock exchange) should result in the same orderbook it crashed with minus whatever we lost during the crash.
After we can do that reliably, then I will strongly consider making an advanced mode level where a crash / market reset / rollback of trades is a planned feature of the level (they are absolutely a thing which happens in real life) and dealing with it from the player's perspective is the main task. (e.g. What would you do if you got a message from the stock exchange saying, basically, "We lost data. How much? Umm, not sure, the last orderId we have is X. Orders after X are no longer on our books but, um, I guess there exists the possibility that they will nonetheless settle successfully if they matched before we went down. ... Good luck to you!")
I imagine you've read it, but if you haven't, you may want to read about the LMAX architecture (from which they created the well-known Disruptor) because they dealt with this exact same problem: http://martinfowler.com/articles/lmax.html
For other readers who aren't aware of that pattern, it's called Event Sourcing. (currently implementing it using Firebase as a combined message bus + storage)
Please don't trigger a kernel panic in one of the stock exchanges; you're on a server with many other actual humans even if they're not hooked up to your stock exchange. More guidance on "bugs you should not attempt to exploit" versus "Things which are totally in scope" at a later date -- Thomas is writing it.
What happens if your process crashes? how do you rebuild the order book?
Great work btw guys, can't wait to get started!