Each game "Server" for SimCity is actually an Amazon EC2 cluster of servers, with 1 central master DB server. Even when the servers were "full" on game launch, all of the EC2 servers were responding to requests normally - it was the cluster's master DB server that was slow. All of the "servers" are actually in the UK Amazon EC2.
This brings us to the scalability problems and why regions/cities are not shared to all servers. The database is the bottleneck, so sharing regions between servers would only worsen performance.
That makes it even more baffling why they couldn't bring up more servers.
If it's all just a chef/puppet based infrastructure in EC2, you should be maybe 20-30 minutes away from pumping out a new 'server'. One is as easy as ten, at that point.
We're talking about EA here. You need to include the latency required to go through enough bureaucracy layers to approve the expenditure of funds on another cluster.
Not to say that there isn't bureaucracy in a company their size, but the launch of a game this size is a really big deal. There's a tremendous of PR, and they're getting scathed. Polygon downgraded their review of 9.5 down to 4.0 because of the server issues. I think if they could cut a decent size check to fix the issues, they would. The problem is likely in engineering, like a database that isn't scaling.
> The problem is likely in engineering, like a database that isn't scaling.
If it were that simple, just cut the number of users per cluster and throw 10 more up.
> I think if they could cut a decent size check to fix the issues
Doubtful within the context of a quick fix, but it is likely the root issue. See above simple solution that takes 30 minutes to roll out. EA is not a company run by engineers, its not a company run by people that understand anything about engineering. What sounds like a simple solution to us that can easily be implemented by throwing money at it and reaping the customer goodwill is completely foreign to a company like that. You may as well be speaking Klingon when you make the recommendation to just throw new clusters at it.
I've met enough people who worked at EA to know they have competent engineers and managers -- they are not completely inept. When hundreds of millions of dollars are suddenly at risk, you have a clear channel straight to the CEO to get the resources that you need.
I'm giving EA the benefit of the doubt that they ruled out 30-minute fixes. I can't see how any of us can really speculate as to how long it should take to fix when we don't really know any details. For example, if it was a database bottleneck, would you commit to walking in and fixing it in 30 minutes? Or even 30 hours? I think you'd want to know the details, because the scope can easily be off by 1-2 orders of magnitude.
> I've met enough people who worked at EA to know they have competent engineers and managers -- they are not completely inept.
Nope, nope, nope. Nobody in their right mind will believe you that. If they fabricate such a game launch then they are complete morons. You simply can not claim that they are competent after that clusterfk.
Under "competent" I understand "Having sufficient skill, knowledge, ability, or qualifications." (definition from wiktionary). Their engineers clearly have neither the skill nor ability to fix the problems in timely manner. Nor did (or do) they have enough knowledge how the servers behave under such high load, otherwise they'd fix it before the launch. And if the problems were known, then the managers failed to delay the launch. A company which knowingly releases such a game is not run by competent people.
I believe it. All it takes is one higher up executive to pick a launch date and mandate that it happens, damn the torpedoes. This says nothing about the competence of the engineers and their managers in the trenches.
I mean, you could question their willingness to work at the company. But, working on the latest sim city sounds like a fun project and people like paychecks.
Bingo! As a former EA senior engineer, my money is on someone in EA marketing setting the dates and forcing an 80-hour week crunch time for the duration of this title's production. They probably even brought in "experts" who ignored foundation code and slammed in their own logic "'cause it works" and now they have a clusterfuck of logic no one understands. Been there, been screwed by their managers, and left for good. EA can rot in hell.
Databases many times are the bottlenecks of infrastructure. You can't just spin up new instances easily to fix the problem. Many times you have to completely re-architect your database schema and architecture to handle to increased demand .
This brings us to the scalability problems and why regions/cities are not shared to all servers. The database is the bottleneck, so sharing regions between servers would only worsen performance.