At that moment, we can't recover from that and players connected to one of the crashed exchanges (1 server process: N exchanges) are told to restart levels, which is not an ideal outcome but it's essentially free. I hope to eventually be able to do something like use our persisted-out-of-memory (e.g. to NSQ) tapes of the orders, bring up the crashed exchanges, and fire every order back into it in order, which (since it's a stock exchange) should result in the same orderbook it crashed with minus whatever we lost during the crash.
After we can do that reliably, then I will strongly consider making an advanced mode level where a crash / market reset / rollback of trades is a planned feature of the level (they are absolutely a thing which happens in real life) and dealing with it from the player's perspective is the main task. (e.g. What would you do if you got a message from the stock exchange saying, basically, "We lost data. How much? Umm, not sure, the last orderId we have is X. Orders after X are no longer on our books but, um, I guess there exists the possibility that they will nonetheless settle successfully if they matched before we went down. ... Good luck to you!")
I imagine you've read it, but if you haven't, you may want to read about the LMAX architecture (from which they created the well-known Disruptor) because they dealt with this exact same problem: http://martinfowler.com/articles/lmax.html
For other readers who aren't aware of that pattern, it's called Event Sourcing. (currently implementing it using Firebase as a combined message bus + storage)
After we can do that reliably, then I will strongly consider making an advanced mode level where a crash / market reset / rollback of trades is a planned feature of the level (they are absolutely a thing which happens in real life) and dealing with it from the player's perspective is the main task. (e.g. What would you do if you got a message from the stock exchange saying, basically, "We lost data. How much? Umm, not sure, the last orderId we have is X. Orders after X are no longer on our books but, um, I guess there exists the possibility that they will nonetheless settle successfully if they matched before we went down. ... Good luck to you!")