The next morning, my girlfriend and I ventured out on foot to attend a yoga class. The yoga studio was closed, so we went to the Waffle House next door and got something to eat. They told us that their water supply had been compromised, but had made special arrangements so they could make coffee safely. We enjoyed a hearty breakfast.
We learned only later that some areas of town had gotten two feet of rain, that houses and businesses were underwater, that multiple dams had been breached, that we should boil all our tap water for a week, and that most people in town were without water entirely. My employer (the University of South Carolina) shut down for a week.
I suppose it wasn't the Waffle House by Garner's Ferry Road and I77? I have relatives near that part of Columbia and was supposed to see them that early October but wasn't able to due to that massive amount of flooding. It was unreal. Luckily, they love on a hill and avoided a lot of damage.
I am a huge fan of Waffle House. I love their hash browns. There are none on the West Coast. It is one of the few things I miss about the East Coast.
I am incredibly impressed. They tend to look like glass boxes,* typically with at least two outer walls of large windows. One back wall is bricked in on the kitchen side and one end wall where the bathrooms are. The front end may have glass on three sides. This is where extra seating goes.
They typically have a galley kitchen with bar seating in the middle and booths to either side that are readily served by cooks and wait staff, plus that additional seating area towards the front (towards the main road, usually).
They have sort of a bad reputation as a dive restaurant, probably in part because the building looks so much like a glorified trailer and the prices are low, but the food is good and I have always loved the chain. My dad grew up on a farm, so I come from humble people who like down to earth places like this. Reading this article makes me feel really good about being a huge fan, in spite of the classist contempt some people have for the chain.
You would think that a place that (very often) looks like a glass trailer would not be the first thing open after a hurricane. So, I am astonished, but happily so.
Anyone that eats at a place based on the decor or the other clients is missing out on a lot of good living. This goes up and down, for Waffle House and the Savile'. Generally, if people decide to eat at a place based on what they think others will think of them, then I think you know what to think of them.
That's something I've noticed as well. My own take on Mexican restaurants is, the more elaborate the decor, the less impressive the food. The little taqueria in the strip mall has far better food at a far lower price than the impressive-looking chain restaurant, in my experience.
It's called : Yellow Sign Mexican, at least in LA. All the best Mexican food places tend to be in a strip mall and have a yellow sign with black letters on the 'billboard' thingy for the strip mall
The real testament to Waffle House is the way more than a few hipsters of my acquaintance started going for a serving of post-party irony--then found out they really love hash browns in some variation of Scattered, Smothered, Covered, Chunked & Topped. Even more especially, it's the the vibe: Waffle House is one place pretense won't get you anywhere, so just chill with your crew and confess your sins as the sun comes up.
Bingo. You can't pretend to be anything in a Waffle House. You are hungover, we know it, you know it. Who cares if you drove up in a Bugatti? You're still eating dollar fried eggs and dollar coffee just like the rest of us, you just can't showboat.
Indeed, one of my favourite restaurants in the world is BBQ King in Sydney (Liverpool St). Mind you it has been 15 years since I last ate there but the décor was always 20 years behind, serviceable and well used. The clientèle was always broad with a skew towards the low key. But what most people didn't realise is that quite often you would find chefs from Sydney's Michelin establishments eating there. The food was to die for; their BBQ sauce was a bottle full of smokey heaven. What's more the prices were very very reasonable. Indeed too many people miss out on some of the wonders of this world because they don't want to be associated with "that type" (whatever "that type" maybe).
In Australia it may be different, as I know it tends to be a bit more classist than the US. But, yeah. I used to know one of the best bartenders in New Orleans, or so he said he was, I think maybe they all were after your 5th Hurricane. His favorite place was out in the swamp with a single string of Christmas lights as the only illumination source for the whole place. They had free drinks on Tuesdays if you showed up naked, all day, but shut that down when only men showed up, all flabby, sweaty, and plastered. Still, my friend liked that place better than anything in 5 miles of Bourbon St. and said they had better drinks too. It's about what you want out of it. Good food and drinks, or the Instagram shot?
I grew up in a suburb just off Buena Vista in Columbus, GA. The Waffle House on Buena Vista was "my" Waffle House.
It was in that parking lot that some stranger tried to buy the family car from my parents. My parents bought it new and kept it for more than two decades. It had become a "classic" and was still in good shape.
Lots of good memories.
And I love their decor. It is practical and convenient. The layout is extremely efficient. If you know what you want, you can be served hot food in minutes.
I don't know why there aren't any other chains trying to imitate parts of their model.
>If you know what you want, you can be served hot food in minutes.
>I don't know why there aren't any other chains trying to imitate parts of their model.
I've noticed this too. It seems like there's a split between "fast food" and "sit down" in the chain l
world, bit no one else goes for the short order diner style. It's really a shame, as I think there's a big space for it.
What you're talking about is, in more modern times, called "fast casual". It's not quite fast enough to be fast food, because it's actual real food, but it's cooked in an efficient enough process that it can be served within minutes.
Yet those places are 4x the price of Waffle House. I mean, yes, you are paying for the decor and the avo-toast, but they are still a lot more than dollar coffee and 2$ hash browns.
I'm willing to bet every town with over 500 people in America has a shitty local diner with greasy food for dirt cheap prices, and any town over 10k has at least one Denny's, so "why doesn't anyone do what Waffle House does" isn't really the question. The answer to that is "plenty of places do", they're just not chains. But really at that price point and that kind of food, if you really need a chain restaurant, McDonalds is going to fill that niche just fine.
The other places are more upmarket, yes, because "greasy food for really cheap prices" is literally McDonalds. Anything between that and a real sit-down-and-order restaurant is called fast casual and you're going to pay a bit more for it.
I don't think there is an official Waffle House in NY -- I don't see it on their locations map; and googling for Waffle House Brooklyn returns some random local establishments. Would love to be wrong if you have a ___location for it!
That's a lovely study in contingency planning. This is something that always strikes me when doing research in line of my work, how many companies are simply totally unprepared for even the most obvious things that can go wrong. It's logical, paying attention to the happy path is where the business is, it is where you grow. But if you only concentrate on the happy path you're a small step away from a disaster.
Now, natural disasters such as this one are much harder to plan for and deal with than the issues that could affect your average start-up. When you're moving and selling atoms the chain from raw materials to revenues is a very thin one with many weak links just waiting for the right conditions so they can break.
In contrast with that, in IT most disasters are man made and easily protected against. And yet, few companies do.
I get the feeling that national disasters are actually the easy thing to anticipate. If you are on the coast, hurricane or flooding. North Dakota is tornadoes and flooding in river basins. Fire is an everyone problem.
Its the weird crap that really gets your goat. I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement. Or when the primary, cannot fail contingency fails. A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on.
I actually got wrote up[1] as being "paranoid" in my backup strategy for a government grant we had in the 90s. The joke of it was that the site visitor who wrote me up was a project director at another site. That site experienced a flood and required extensive efforts to recreate their data. I was young, and a bit vengeful and might have made some comments during the next national meetup.
Sometimes, a lot of disaster recovery and contingency planning is politics above the level of the people who should know better.
1) They thought it was a problem during a site visit and we had to write a response that had to get approved.
A place I once worked had their servers in-house. They had UPS on the roof, duel everything (fibre links etc). Apparently there wasn't a single point of failure in the system.
One day, the servers just vanish. We go into the server room to check. Someone had plugged all the machines into the same multi box / power extension cable and the fuse in it had popped.
Sometimes all your planning can be undone - especially once humans are involved :-)
> Apparently there wasn't a single point of failure in the system.
Redundant power supplies on alternate phases is a pretty common element in designing any kind of server setup. So this wasn't a setup with a 'single point of failure', it was a setup with a very explicit single point of failure (several, in fact).
Someone linked to this excellent slide deck yesterday:
I remember a data center that supposedly had dual fibre links to the outside. Found out later (after someone cut the fibre) that they actually linked back up just outside of town. I understand lawyers were needed.
'Fully Diverse Paths' is something we take great pains to verify with our WAN provider, for exactly that reason.
Because it is said that if you are heading into the woods and worried you might get lost, take a length of fiber with you. If you get lost, simply bury the fiber in the ground. In five minutes a man in hi-viz with a backhoe will come along and dig it up.
A certain important data center near here has dual fibre links, as mandated by their redundancy requirements. They're installed along both sides of the same major road.
That's not so bad. The typical mode of failure for fibre lines is not flooding or tornadoes or landslides or anything else that would affect both sides of the road at the same time. It's someone digging up the road, which is unlikely to happen on both sides simultaneously.
Heh. The story i recall their initial problem was that the failure as always after office hours, but they could not find any fault with the hardware or software.
So as a last desperate attempt, the person telling it basically sat watch inside the server room and noticed the door opening, a arm reaching in, unplugging a socket near the door, and plugging in a cable from the hallway.
The very next day security was switching the lock on that door.
> I get the feeling that national disasters are actually the easy thing to anticipate.
Their presence, yes. But the frequency and intensity are hard to put any bounds on.
> I remember finding out the power main to our current place is located next to a T-section of road. Slide forward too far and hit the power main. Brilliant placement.
Something exactly like that happened with a gas line in Slupsk, Poland. The effects were incredible.
> A place I once worked at had a >$1 Million UPS (generator / batteries lasting weeks) for their data center. Semi hit the power main, and the UPS failed to turn on.
Just having a generator is not enough, you need batteries for the ride-through period and you need to do periodic tests. Maybe they did and it still didn't turn on?
I'd take being called paranoid as a badge of honor, better to be paranoid and up and running than down and searching the yellow pages for data recovery services.
Oh, they had the whole line. UPS in the server room, with batteries and diesel engine outside. Tested quarterly, I was told. In server room batteries did their job, but the stuff outside didn't. Took 4 hours to restore power and another 32 hours to restart the servers and verify their data.
[edit: about the site visit calling me paranoid] I was a bit ticked off, but I guess someone who didn't even do the basics, as we later found out, would not get the need.
Ok. Even a small DC that I'm intimately familiar with has a back-up diesel that they test religiously on the 1st of every month, they have the whole building running on the genset for an hour or two and then they shut it all down again. Iirc it failed to start twice in the years that this setup has been there, which means the chances of a problem will never be quite '0'.
When I ran a gas station in North America one of the first things I did was install a diesel generator. My reasoning was that with on average 50K liters of diesel in the underground tanks and the genset powering the pumps it should be good for most problems that we'd face. And in fact it was, which we managed to prove beyond any doubt when all of the North-West of the US and a good chunk of Canada went without power for a couple of days within a year of installing that genset.
Sometimes being prepared is a waste of money and time in retrospect, but it helps you sleep well, and sometimes it is the difference between a community continuing to function or people dying.
The classic Bell System standard was that generators were started once a month and run for several hours. Once a year, each central office ran on emergency power for 24 hours, with external power shut off.
In the entire history of the Bell System prior to electronic switching, no central office was ever offline for more than 30 minutes for any reason other than a natural disaster.
Of course I've known that a DR plan isn't valid until you've tested it against your production environment, but I'd never heard that Bell System story before. It's inspirational - run 24 hours in emergency mode each year, with the normal way turned off/disconnected, don't just do failover/failback.
> down and searching the yellow pages for data recovery services.
Back when I was a medical tech, we lost a hard drive with patient data on it. I ran a chkdsk with no luck, and out of ideas, called a data recovery company. They stated it was $3-5k to attempt to recover, and asked if we'd done anything to the drive. "Just a chkdsk, nothing else". The guy on the other end of the phone was genuinely annoyed by this. "I wish you people wouldn't do that" delivered with that tone. Yes, this guy's position was that if you ever have a disk problem, your first port of call is to drop $3k+ at a data recovery place, don't even attempt to do some preliminary checking.
I can see his point though. From his point of view, by the time the disks arrive on his desk anything that tried to repair the damage has likely made it worse and hence his job that much harder. But of course $3-5k is sufficient money that most companies will make the first attempt themselves, especially since the alternative does not make any guarantees and will cost money either way.
I've had to do some data recovery during the course of my personal life. If the data is valuable, the #1 rule is "always run a dd backup to another device first" IMO. Then you can play around on a device that you know isn't on the brink of failing.
Years ago we had a small, in house, departmental server room at Stanford. It was by no means industrial strength but we did our best on a limited budget (security, climate control, battery backups, disaster recovery procedures, etc). One night we all got pinged about impending server failures with 30 minutes to respond before the UPS ran out of juice. Turns out the cleaning crew needed an outlet for their vacuum cleaner and decided to unplug one rack of servers to finish up their shift. This episode reminds me of a favorite haiku:
I'm also a volunteer firefighter. Last week I was at a brush fire in a semi urban area that got so bad so quickly that it prompted three choppers and the first time in this half of the state that a DC10 VLAT (very large aerial tanker) dropping retardant was deployed due to the threat to multiple residential and commercial structures, several of which were still lost.
I remember the first time I saw a fire start from a discarded cigarette. The sight was awesome to behold, especially as it burned uphill on one side of a freeway.
It's why I'm extremely paranoid about dousing anything that was once on fire.
All it takes to start a fire is an idiot and a cigarette.
Last I checked, everybody's data center or building can catch fire or have a fire inside. I would love to see the building that has computer equipment in it that cannot catch on fire. You are one bad UPS / power supply away from a rack fire. Never mind the electrical panel itself.
And every place even those listed on the map can get them unless you build your data center on some terrain with absolutely no plan life around. You need to deal with fire outbreaks even if you are not on the map.
And even then there is lightning induced fire. Plenty of natural fires start that way and there is no reason why your building can't be the starting point of one. Though fortunately most DCs take lightning protection quite serious and are probably amongst the harder buildings to ignite.
I read an excellent article on Sear's logistics after it was mentioned here. The parallels between keeping products flowing and data centers operating was quite striking. Multiple suppliers for key material, multiple transportation technologies, a series of plans which would keep you available as you fixed an underlying problem (defense in depth).
It is a pattern or process that repeats itself in everything from keeping web services online, stores stocked, casinos from being robbed, and even individuals from being dumped into poverty by events. I expect if there was a book "Everyday Prepping" it might actually sell pretty well :-)
My take away from the article however is that it is better to choose indicators that have a wide dynamic range :-).
> In contrast with that, in IT most disasters are man made and easily protected against. And yet, few companies do.
If I'm competing in a high growth industry against somebody who is busy planning for the worst, I'll move twice as fast and put them out of business before their plan can matter.
And this is why Silicon Valley isn't always trusted.
Yes, you probably can, and more often than not your gamble will work. You'll move fast and disrupt and break stuff and odds are you can even say, up front, 'I'm moving fast to give you more without that boring planning stuff!' and may still be able to put the other guys out of business: depends on how well they handle it, and how hard you can push.
And then the natural disaster strikes and you're moving fast again, to make an insurance claim (or get out of town!) and the people who in turn gambled on you, are boned. Perhaps very catastrophically boned. And it's their fault, because let's say they were told up front that it was a gamble.
But people are dumb.
And you can put a lot of more responsible community oriented business-operators out of business before their planning matters, because people are dumb.
Depending on what you do (taco stands? Hospitals? Building dams and bridges?), your very Silicon Valley approach could be anything from colorful to criminal. I will hope the bridges I drive on are 'planned for the worst'. But there's only the inertia of social expectation, and whatever exists in terms of enforceable legislation and policing, to ensure that. And you can always violate legislation as part of your 'move faster' strategy, and expect to just deal with it somehow later. So that leaves only the inertia of social expectation.
> I'll move twice as fast and put them out of business before their plan can matter.
Yes, maybe. Or you'll be put out of business because you did not have a plan.
You could attempt to justify your stance by saying that in the growth phase of the company nothing else matters but the reality is that you can't really retro-fit the attitude and set-up that your future company will require once you get there if you didn't do this from the beginning.
See, you can overdo anything, including security and contingency planning. But that isn't what this is about. It's about knowing that you are taking a conscious risk, knowing what the consequences could be and knowing that if those risks do materialize what you will do.
If some of those paths end with 'game over' then so be it. But there is no excuse for not knowing and it is exactly those things that you did not know about that will kill you.
So much good luck with moving twice as fast, headlong into a wall that you could have seen coming.
Enough that I get called out at least once per year to do a rescue attempt. Good for me: they usually work. Bad for the users: they don't always work and they're not always perfect. Of course all of these companies retrospectively agree that maybe they should have had their back-up setup tested a bit better. But that's hind sight, and I'm quite convinced that prior to failure there isn't an argument in the world that would have swayed them to actually do it.
And so the cycle continues. I probably shouldn't work too hard to educate people about this since it negatively affects my bottom line but those very same people definitely carry insurance on their real estate and their vehicles.
A backup that you haven't tested (and with a test I mean to rebuild your business from the ground up using nothing but the backups) is most likely incomplete.
Bonus points if you remember to store the decryption key outside of your backups ;)
Not only are you unnecessarily rude to a very intelligent community member, but you demonstrate a complete lack of respect for your customers. Please change your attitude.
Only if it is a winner takes all industry, and only if nothing bad happens before they go out of business.
Lots of industries have clients that actually care about the robustness of their service providers as well; if in the future, once something invariably does go wrong, you have effectively just signed your own death warrant if you can't recover quickly enough.
What if the plan b is needed? How are tech companies with data in the hurricane doing right this second? That really depends on their planning doesn't it?
Or put another way, a good Plan B puts makes your Plan A possible in some situation that would otherwise mean failure.
Depends on the company too. A lot of the time this sort of planning requires driving from employees, but in most natural disaster scenarios what happens to the employer is the least of their concerns. The type of person who does plan for things like this is likely buying tinned food and bottled water, not thinking about being able to log in to work remotely.
The other thing is, it simply doesn't matter for most companies. They'll be out of commission for a week anyway because most of their employees are. There's no point have their online storefront online while the warehouse is have wheelbarrows of mud wheeled out.
As long as the data is backed up and recoverable, that's enough for most.
That's my favorite Waffle House[1] (It was reconstructed after Hurricane Katrina)! I used to eat there all the time and watch the shrimping boats out on the gulf.
This comment doesn't have much purpose other than one of those small world moments.
While not in the food industry; I work for a national auto parts supplier which of course has more than just auto parts; large scale natural disasters are something to behold. From the store teams who check both company and privately owned stores for needs to staging the warehouse with goods the local community will need as well as the warehouse as well; that can be as little as a semi trailer based generator.
still as with all organizations what also comes up is spinning off demand to other stores and warehouses. getting those outside the affected area ready to help those affected and so on.
the key though is communication and you must develop a well defined plan of how to communicate. this means from who, how often, the methods of delivery, and with email it can me using specific templates to make it readily apparent everyone is on board.
anecdotal, with family and friends of who volunteer for fema, there are just "constants" that they also use for knowing when things are getting better, some are restaurants and other are simply commodity foods and goods.
The mention of the Waffle House Index reminds me of the Big Mac Index used to compare currencies. The idea is that a Big Mac, being something of commodity made from common ingredients, should cost the same everywhere in the world. If it is significantly more expensive or cheaper in another currency, than that currency must be over or under valued.
Waffle House has a map of their locations, but it doesn't show which ones are down. Waffle House #1260, on Galveston Island one block from the beach, is probably in the worst position.
I was living in Udon Thani in northeast Thailand in the year of the great Bangkok floods.which is well over a ten hour drive from Bangkok. We had heavy rains but no real flooding. But when Bangkok was hit, a huge part of the supply chain for the whole country was trashed. The western style supermarkets were almost empty for months after the floods. It was impossible to buy basic things like instant noodles and disposable diapers for months. But fresh food, vegetables and meat were mostly from local sources so the wet markets where you traditionally shop every day for food were largely unaffected. It was the "modern" infrastructure which shown to be little more than a house of cards.
I would imagine that in so-called developed economies, there isn't anything except the "modern" supply chains. If a supply chain is using terms like "just-in-time" then there is no way of dealing with things when infrastructure goes down.
If the natural disaster had actually hit your area the local sources would have been destroyed and unavailable for months while the externally supplied sources would resume being supplied as soon as basic local infrastructure was repaired.
> But when Bangkok was hit, a huge part of the supply chain for the whole country was trashed. The western style supermarkets were almost empty for months after the floods. It was impossible to buy basic things like instant noodles and disposable diapers for months.
Don't even begin to compare Developed Western countries infrastructure and supply chains to anything in Thailand. To compare the two is laughable at best or shows your bias at worse.
Here where I live there are both supermarkets with modern supply chains as well as roadside stands and small markets who buy local, so we're not particularly screwed by our Western society. About 30 miles from me is a valley that supplies tons of produce during the spring and summer. They don't grow tomatoes or peppers, but I can source all of my other ingredients locally. Winter would be troublesome but I've usually got enough dry goods to get by for a few weeks.
Except infrastructure goes down all the time in developed nations when disasters hit. The difference is our supply chains are up and running within days/weeks not months.
This actually reminds me of one year I was heading home from my folks' on New Year's. As usual, I found myself driving past Michigan City, IN, around or after midnight, with the usual lake-effect snow going full blast. As not-so-usual, I decided I was going to have to pull off for the night, so after checking into the local Super 8, I set out to see what late-night eating establishments might actually be open. Almost nothing. I swear, this was the first and only night I ever found a darkened Denny's. (Oh, my God! They closed Denny's! You bastards!)
I finally found one source of sustenance actually braving the weather that night: a White Castle, probably only because it was attached to a 24-hour gas station. At least I didn't have to go to bed hungry.
I have no idea whether there was a Waffle House in MI City, or where I might have found it if there were, but if all else fails... I guess there's always White Castle.
We were driving away from the Gulf in the teeth of, I think, Ivan back in 2004, the freeways had been converted to only one direction, it's the only time we've ever eaten at Waffle House, they were the only thing open, as we left they didn't look like they were closing
This is very interesting and mostly true but I think I speak for all coastal Texans when I say it's not really a good measure of how we're doing, especially those of us that have lived through several hurricanes.
https://en.wikipedia.org/wiki/October_2015_North_American_st...
The next morning, my girlfriend and I ventured out on foot to attend a yoga class. The yoga studio was closed, so we went to the Waffle House next door and got something to eat. They told us that their water supply had been compromised, but had made special arrangements so they could make coffee safely. We enjoyed a hearty breakfast.
We learned only later that some areas of town had gotten two feet of rain, that houses and businesses were underwater, that multiple dams had been breached, that we should boil all our tap water for a week, and that most people in town were without water entirely. My employer (the University of South Carolina) shut down for a week.
Indeed. Waffle House doesn't kid around.