I'm not highly experienced in this realm but I sometimes wonder if the liability in our industry is misaligned which may cause many of these symptoms.
When a company faces a huge security breach due to negligence, who pays the cost? I've known many engineers rightfully pointing out necessary work to prevent security issues that have to negotiate with product owners and business stakeholders to find time to perform such work. And guess who never gets the time to work on those issues?
How many companies actually audit their dependencies and run their own package management infrastructure to protect themselves against liabilities?
If developers were liable then companies would be obligated to hire a professional engineer, pay their insurance, and there would be repercussions for avoiding their recommendations. I suspect companies would be much more cautious about "releasing early, releasing often," and "moving fast and breaking things," if there were penalties for making poor decisions that affect the safety and property of people.
That's all true but I don't think it explains the effect GP is talking about.
Security aside, I think it just comes down to what the business incentives are for different approaches to software development. To develop good software you need small teams of experts with ___domain knowledge and very strong programming skills. But good software isn't the only kind of profitable software.
A lot of big businesses are working with multipliers on the benefit side that eclipse the factors on the cost side. If you don't have a particular piece of software 'X', where X is something that would generate revenue multiplied across a really large customer base, then almost any strategy that produces X is going to be a strong one. From a business perspective it's not going to make any difference if you hire a 5-man black team to pump out the perfect bit of software or 150 cheap consultants to hack together a giant pile of shit. The former might make you $150M instead of $120M, but nobody is going to have stats on the path not taken to compare to in retrospect, so there's roughly zero market pressure to develop software properly.
Then there's two factors that explain why businesses don't take the better approach. The first is that every big business I've worked for has had layers of management and departments, where budgets are allocated artificially and aren't perfectly tied to results like they would in a pure software business with flat management that's reinvesting profits into their flagship product. That creates an environment where the objectives for middle management aren't perfectly tied to the objectives for the business (e.g. increasing their team's head count at the expense of their software).
The second is that having a larger team mitigates risk. If you put together a small team of crack developers to build a perfect product there's somewhat of a chance that they fail. Things like turnover and personal circumstances affect you a lot more. The swarm of average contractors doesn't have as big of an upside, but they're much more likely to get you to 80% of the way there, so if that's indistinguishable to a perfect solution (as above) then there's this constant natural pressure to overhire and reduce the quality of the team to the lowest common denominator. Big businesses love treating their devs as cogs, even if all the literature on the engineering side suggests that that approach doesn't work.
Basically, I don't think it's just a case of 'managers are idiots and don't know how to build good software', businesses build crap software because it works and it's profitable. But it also creates constant opportunities for startups to eat their lunch, and the worse the economy gets, the better your software has to be (because the cost side of the equation starts to matter more).
When a company faces a huge security breach due to negligence, who pays the cost? I've known many engineers rightfully pointing out necessary work to prevent security issues that have to negotiate with product owners and business stakeholders to find time to perform such work. And guess who never gets the time to work on those issues?
How many companies actually audit their dependencies and run their own package management infrastructure to protect themselves against liabilities?
If developers were liable then companies would be obligated to hire a professional engineer, pay their insurance, and there would be repercussions for avoiding their recommendations. I suspect companies would be much more cautious about "releasing early, releasing often," and "moving fast and breaking things," if there were penalties for making poor decisions that affect the safety and property of people.