1. Seeing my boss just completely side step an issue, where his switch was sending vlan tags to his carrier causing the carrier to shut down the port, by purchasing a switch that didnt support vlan tagging. I died on the inside but lesson about end user troubleshooting learned.
2. Being able to break into a 24 hour data centre because the data centres building management just didnt give a shit. Turns out they had fired all their staff and were leaving the DC empty until they liquidated. We managed to get our kit and bail. But like physical security provided by a third party is ephemeral.
3. I used to lock my bike to my tool shelf. A thief came along and used my hacksaw from my toolshelf to break my bike lock and steal my bike. Thieves will use anything available, not just what I want them to use.
4. Had 2 console windows open. First to a Junos device on my desk. Second to a Junos core router. I reloaded one, and didnt realise I had reloaded the wrong one until we lost an entire state. Multitasking invites risk.
5. Customer had a huge failure. Kept insisting they would rebuild completely, but would routinely beg for just one more dodgy workaround just to keep them going. They never rebuilt. That network is perma fucked. Customer promises are worthless.
6. Much like 5. Customer got bought out by a larger firm. Began to raise new deployments by escalating to the after hours noc, pretending that new deployments were actually failed existing sites. We asked them to stop and they never did. If theres a large enough financial interest customers will just ignore contract and good practice to make the payday.
My customer was a mid sized ISP. They contracted us in to replace a network engineer who was leaving who they couldnt replace. He wouldnt say anything about why he was leaving, other than he wanted to spend time with his newborn. Fair enough.
Soon after the entire C level resigned. The new C level execs immediately went wtf.
The business had:
1. Sold to an investor, the former owners becoming the C level.
2. Had an agreement that they would stay on for x years and deliver xyz revenue before they got paid for the business.
3. They had been selling services with massive install fees, and then not delivering those projects. Sales in this manner had kept the book completely green. What they had delivered didnt meet spec. They just massaged relationships to pretend it didnt happen. They also opposed any kind of internal documentation of these problems.
4. They had 15 or so of these projects in the pipeline when they left the business, right on the 3 year mark. Most of them were undeliverable for one reason or another.
So what happened was that the incoming C level guys had to basically deliver on a bunch of over promised contracts, without the build costs in the bank, while also fending off a business owner who didnt acknowledge the problem, and customers with existing issues who were quite angry.
They lasted ~3 months before they quit too.
Lesson: No business is big enough to avoid the consequences of its actions. A business can look good even on the books but have the "Find out" in the future. Nothing in business is ever settled or guaranteed.
Also:
Being a contractor means largely being insulated from these issues in a way that allows you to observe things falling apart. Just dont fall in love with the customer.
> 3. I used to lock my bike to my tool shelf. A thief came along and used my hacksaw from my toolshelf
You locked your bike to a toolshelf that contained the tool they used to steal the bike? That would be like putting jewelry behind a glass cabinet and putting a hammer next to it.
2. Being able to break into a 24 hour data centre because the data centres building management just didnt give a shit. Turns out they had fired all their staff and were leaving the DC empty until they liquidated. We managed to get our kit and bail. But like physical security provided by a third party is ephemeral.
3. I used to lock my bike to my tool shelf. A thief came along and used my hacksaw from my toolshelf to break my bike lock and steal my bike. Thieves will use anything available, not just what I want them to use.
4. Had 2 console windows open. First to a Junos device on my desk. Second to a Junos core router. I reloaded one, and didnt realise I had reloaded the wrong one until we lost an entire state. Multitasking invites risk.
5. Customer had a huge failure. Kept insisting they would rebuild completely, but would routinely beg for just one more dodgy workaround just to keep them going. They never rebuilt. That network is perma fucked. Customer promises are worthless.
6. Much like 5. Customer got bought out by a larger firm. Began to raise new deployments by escalating to the after hours noc, pretending that new deployments were actually failed existing sites. We asked them to stop and they never did. If theres a large enough financial interest customers will just ignore contract and good practice to make the payday.