I feel these organizations that have a process that prevents critical fixes have a broken process... you either have to be ok with having your servers compromised eg data stolen or leaking user data or you have to be ok accepting that sometimes the engineer fixing a bug, adding a new feature might mess something up. I am inclined to believe a bit more to the side of move fast break things is bette than move so slow you get pwned... but sort of a delicate balancing act...
I studied IT security quite a lot, and implement Windows patches for dozens of companies. While you are technically right, Microsoft releases broken patches _constantly_. If we pushed out every single patch the moment they were released, we would constantly be down and fighting fires. Most small and mid-sized companies don't have hacking campaigns run against them most times. Given this, it just doesn't make sense to push out every single patch immediately. Microsoft's patches are a whole lot more stable when they're a couple months old.
This has been a real problem again in the Windows 10 era. By around 2008, Microsoft seemed to have finally gotten their patch process cleaned up to the point that if you were only taking security patches, they generally installed cleanly and mostly didn't break random things. By about 2016 this has backslid and now Windows 10 seems intent on large scale combined updates and constant servicing stack updates that with undocumented consequences.
It's been a giant pain having spent years trying to get organizations to accept the need and learn to do this stuff reliably only to have the primary source of misery (Microsoft) repeatedly start biting them in the ass again for what should be best practices.
Meanwhile in the same timeframe most BSD and Linux releases have not only gotten their core software updates down to a science, they've also managed to build workflows that can include huge swathes of 3rd party open source and commercial software, which is so hilariously awful on windows that multiple companies build businesses around doing it.
> "This has been a real problem again in the Windows 10 era. By around 2008, Microsoft seemed to have finally gotten their patch process cleaned up to the point that if you were only taking security patches, they generally installed cleanly and mostly didn't break random things. By about 2016 this has backslid and now Windows 10 seems intent on large scale combined updates and constant servicing stack updates that with undocumented consequences."
Microsoft laid off all their QA staff in 2014, so it's hardly surprising. If anything, it's a wonder that it's not much, much worse than it is now.
It’s partly broken process - my point being that the people at the top are more to blame than the sysadmin - but also that this is more expensive than people like to admit. You either need to accept lower security/reliability or spend more on staff, capacity, and licenses. Lots of places try to cut that corner and it’ll seem to work until, as Warren Buffet likes to say, the tide goes out.
This is a really tricky problem in government because the pay scales can be very hard to change. For example, the U.S. federal scale has hard caps - the GS scale max is currently $170k, which might not sound that bad but historically the higher-level positions were senior and relatively limited, so it’s not like you can just effortlessly bump all of your developer positions up to the highest grade without hitting budget caps and other people being upset that someone outside of IT needed 25 years of experience and managing a bunch of people to get to the same rank as you’re proposing to offer to non-entry level developers. That probably means you’re hiring people at lower levels which are more like entry level pay.
A few years back they actually had to try to have a chance of hiring good infosec people but that requires a lot of political wrangling even if everyone agrees that it’s a good idea. (I know someone who got tired of waiting - jumped to a well known tech company for a cool 200% raise)
Many vulnerable organizations do not have "engineers who fix bugs", they have teams of accountants and bookkeepers who run Excel and Xero, or teams of lawyers and paralegals who run Word, or medical practices or marketing firms or chemical wholesalers, or or or... The nearest thing most of them have to "an IT department" is the admin person who liaises with their outsourced IT provider and the manager who signs of on the bills every month.
Isn’t this the sort of issue where Defense in Depth comes in? You don’t want to rely on a secure LAN, but having a secure LAN _and_ a hardened server reduces your attack surface in the case of a 0-day
