"Changing one line of code took 6 days!" is a pretty common WTF story. It comes up every so often, and we all get a chuckle at the ponderous rate of change in large organizations.
However, "Changing one line of code broke everything for 6 days!" is also a common WTF story. As a rule of thumb, you can't avoid both of them, because avoiding one entails the other. I wouldn't fault a company that chooses the first story over the second.
The problem is that things that scale up don't always scale down. The processes implied aren't that bad, for handling large commits to a large process. They're obviously overkill for small changes. But just because the processes are a negative effect on this change in particular does not mean that the processes are a net negative overall. More flexible processes could also end up being more trouble than they're worth, due to allowing (encouraging) sloppy work.
Or you could just hire responsible people and ask them to use their judgement, you don't have to choose one or the other you can instead apply judgement as to whether this is going to screw shit up or not.
Honestly, though the best way to handle this situation is with a phone call to the CEO/IT Director getting him to write you an email allowing you to override all the checks to get this in ASAP.
Once you're known for clearing things with hire ups, QA will start being reasonable. You have to keep in mind that most people in QA are type C personalities and will be a stickler for inane rules.
For starters, "just hire responsible people" can't be applied retroactively. Not to mention that… most people are responsible, or rather, the road to hell is paved with good intentions.
Secondly, it's an incredibly difficult thing to scale. That works great in startups, but by the time you have things like "HR Directors" you're no longer capable of having insight into the rest of the organization.
There is a lot to be said about how bureaucracies are the formalization of common sense, and a lot more to be said being able to scale processes out such that you don't end up in situations where stakeholders have no visibility into the work performed.
I just feel that the answer is more complicated than "hire A players".
> Or you could just hire responsible people and ask them to use their judgement
If you know a fail-safe way of only hiring responsible people who always have good judgment... then I hope you're being paid millions of dollars in consulting fees, because nobody has really figured that one out in a general way yet!! :)
Also, those people tend to be expensive... sometimes it's far more cost-efficient to hire less-perfect people, but put rules in place to minimize any catastrophic damage they can create, at the expense of efficiency.
Following the rules to the letter also produces bad outcomes.
The issue is not that using judgement is perfect it's that using judgement usually produces a better outcome than blindly following a rule.
Yes, rules should be put in place for the most extraordinary circumstances or places that good judgement is known to fail. The rule has to prevent the bad thing to be effective while also not creating something worse.
In this case, there was someone with basically good judgement, but they were operating from high within their hierarchical framework, which inherently induces delays. In retrospect, Philip, David, or both, should have been copied on the ticket and directly overseen the policy violation that they deliberately set in motion.
Simply ordering policy violations from the throne doesn't work in practice. Either the person who made the decision or a trusted lieutenant who understands what's going on has to see it through to the end, otherwise the bureaucracy in the middle will simply continue on its preprogrammed course.
Just because they're smart doesn't mean they don't make mistakes. If you allow one line changes to pass without review, eventually someone's going to make a breaking one-line change.
However, "Changing one line of code broke everything for 6 days!" is also a common WTF story. As a rule of thumb, you can't avoid both of them, because avoiding one entails the other. I wouldn't fault a company that chooses the first story over the second.
The problem is that things that scale up don't always scale down. The processes implied aren't that bad, for handling large commits to a large process. They're obviously overkill for small changes. But just because the processes are a negative effect on this change in particular does not mean that the processes are a net negative overall. More flexible processes could also end up being more trouble than they're worth, due to allowing (encouraging) sloppy work.