As people mentioned this site is an exception to how to do things, in that PG actively does not care about search engine results. However, for the people who are interested, here's a few ways you can handle a situation like this.
1. If you add the "max-stale" header to your content you can tell your CDN to hold on to things longer in the event of an outage. What happens is that the CDN will look at the cache time you set and check in as normal when that expires, but if it gets back no response or a 503 code (server maintenance status code- if you work with CDNs this is your friend!) it will continue serving the stale content for as long as you tell it.
2. Lets say you're beyond that, or your site is too dynamic. Instead of setting up an error page that responds to everything, setup a redirect with the 302 status code (so crawlers and browsers know it's a temporary thing). Point that redirect at an error page and you're golden. The best part is these types of requests use minimal resources.
What I do is keep a "maintenance" host up at all times that responds to a few domains. It responds to all requests with a redirect that points to an error page that issues the 503 maintenance code. Whenever there's an issue I just point things at that and stop caring while I deal with the problem. I've seen webservers go down for hours without people realizing anything was up, although with dynamic stuff everything is obviously limited. The other benefit to this system is that it makes planned maintenance a hell of a lot easier too.
Oh, another thought- you can use a service like Dyn (their enterprise DNS) or Edgecast to do "failover dns". Basically, if their monitoring systems notice an issue they just change your DNS records for you to point at that maintenance ___domain. You can also trigger it manually for planned things.
For example, I had a hard time telling people that they should use a proper HTTP redirect from their ___domain "example.com" to "www.example.com" (or vice versa), instead of serving the content on both domains. All arguments about standards, best practice etc. were unconvincing. But since Google startet to punish "duplicate content", I never had problems convincing people again.
Search result ranking is an obvious casualty, but it's not the only one. My Firefox start page (as an example) now has a cached but empty version of the HN pages that doesn't really look quite right. Any system that pulls content automatically, like that, or an offline reader, etc. is going to get confused.
It's simply good practice to get the HTTP status codes correct in the ways you outline.
HN was down much longer than I thought it was because the bad pages were stuck in caches. I had to do a force-reload to see that it was back up this morning.
HN was down much LESS time than I thought, because they told browsers that the "Sorry, but we're down" message should stay in cache until 2024! I believed the status report until today, because it didn't occur to me that someone would issue a status report on a temporary site-wide outage and instruct browsers to consider it permanent.
It was also down for much longer than a lot of services (e.g. status checkers) thought, since their only sensible method of working is to inspect the HTTP status. The result being that HN uptime looks better than it really is; of course, this isn't a paid service or something mission-critical, so exaggerating uptime doesn't really make much difference.
same here, the only way i finally got out of bad state was to click on the logo to go to /news
if i navigate to news.ycombinator.com i'm still seeing the "we're down" message in chrome.
This a strange reaction. I agree that perhaps "this is different", but I am hoping I can assume that the next time I see an article on HN, about another website going down, or losing their data, that the HN community shows as much compassion and understanding. In my experience, when not related to the work of pg, there tends to be a lot of "you should have done X and there is no excuse for not doing Y!!"
If I'm being honest, it's off putting to watch community-wide apologist response to what would normally be outrage of poor execution.
Having spent months telling people that email is not reliable and can fail at any minute and must not be relied on, and then having to put up with the fall out when the shitty provider broke something, I can confirm that people get angry when email breaks.
"email is not reliable and can fail at any minute and must not be relied on,"
In what universe? If email were to stop working in most major corporations that I've worked in for the last 8+ years, the company would basically come to a halt.
Email, for many, many companies is the message/workflow bus, and if it stops - communication comes to a halt.
It is, after electricity, and the network, the one essential function in a company in 2013.
Telephones, Photocopiers, Printers can all cease functioning , with little impact in most technologies companies - but not email.
And yet Email is the most failureprone thing out there. It was not meant to be relied upon in the degree it is today. And yes, many companies grinds to a stop as soon as email is down. There has been no big improvements to the reliability email the last 40 years. I mean, what is the date format in email? Any sort of current standard? What date-stamp are most email readers trusting? Ever had an email that says it arrived 1997? I still get those sometimes.
I don't know where you keep coming up with that - Email is relatively easy to make highly available, and an Exchange Server in 2014, configured with even a modicum of skill, will likely keep running smoothly for the next 10 years. It's as close to 5 9s, of availability that you can get on a software system.
"There has been no big improvements to the reliability email the last 40 years. "
That's just silly. Take 5 minutes to read through http://en.wikipedia.org/wiki/Email and you'll see what improvements have been made in the last 40 years.
Deliverability of email relies on a number of factors outside your control.
You also mention competance, which is available in variable quantities. You might be able to keep Exchange 2014 running solidly, but I've seen people doing scary things with MS SBS Server 2000 (and the exchange that comes with that).
You may be sympathetic, but it seems like most people aren't. Most folks start saying what they'd do better and how they could improve the availability in a weekend.
"actively does not care about search engine results. "
Or data -- they tossed a day or so of submissions and comments when restoring from backup, AFAICT. I'm not saying that's a problem; 4chan loses data and works just fine.
That's a poor comparison. Users expect 4chan to lose that same data as part of normal operation anyways, so you can hardly consider it a problem. That's not the case for HN.
1. If you add the "max-stale" header to your content you can tell your CDN to hold on to things longer in the event of an outage. What happens is that the CDN will look at the cache time you set and check in as normal when that expires, but if it gets back no response or a 503 code (server maintenance status code- if you work with CDNs this is your friend!) it will continue serving the stale content for as long as you tell it.
2. Lets say you're beyond that, or your site is too dynamic. Instead of setting up an error page that responds to everything, setup a redirect with the 302 status code (so crawlers and browsers know it's a temporary thing). Point that redirect at an error page and you're golden. The best part is these types of requests use minimal resources.
What I do is keep a "maintenance" host up at all times that responds to a few domains. It responds to all requests with a redirect that points to an error page that issues the 503 maintenance code. Whenever there's an issue I just point things at that and stop caring while I deal with the problem. I've seen webservers go down for hours without people realizing anything was up, although with dynamic stuff everything is obviously limited. The other benefit to this system is that it makes planned maintenance a hell of a lot easier too.
Oh, another thought- you can use a service like Dyn (their enterprise DNS) or Edgecast to do "failover dns". Basically, if their monitoring systems notice an issue they just change your DNS records for you to point at that maintenance ___domain. You can also trigger it manually for planned things.