It is a major problem, but it is the big unsolved problem of software engineering that has resisted a remarkable number of attempts by unreasonably capable people. Under normal conditions the metrics you are talking to boil down to "the dev says we're making progress". The level of formality with which they say that changes, but not the underlying process of how the metric is generated.
The only way to improve on that is for managers to go in to git and check what is happening. Technical managers can get deep into tracking progress. Non-technical managers cannot.
> It is a major problem, but it is the big unsolved problem of software engineering that has resisted a remarkable number of attempts by unreasonably capable people.
That seems like a cop out. Knowing precisely how long a project will take is famously hard, but knowing the rough progress being made towards milestones is absolutely solvable.
Git is the best place to look, but you don't need that level of detail to see new things being shipped.
Everywhere I've worked (from small startups up to Google) it is absolutely the norm to have clear milestones to work towards on at least a quarterly basis.
> knowing the rough progress being made towards milestones is absolutely solvable
Yeah, but the process for knowing that is ask the dev and they will tell you. Or ask the PM who will ask the dev. Or ask a tech lead who will ask the relevant dev. Or read the spreadsheet entry that the dev updated. All roads lead through conversation with the dev working on a feature. There are occasional exceptions, but they are rare.
If the dev is willing to lie, or honestly mistaken, or just too inexperienced to estimate how they are doing then it is close to impossible to gauge progress.
> That seems like a cop out
You would probably believe the number of people who say that.
Yeah. But the dude in the article lied about progress so I have to include it.
> Estimates can be wrong, but I think a battery of test is pretty objective.
Yeah. But if 80% of the tests are passing, is the project 80% of the way through the timeline to completion? No. They are useful because they measure correctness but they don't measure progress.
I also don't think it is reasonable to expect non-technical managers to interpret technical tests; even as a rough measure of progress.
If 80% of the tests are passing, then another week of work get 2 more passing, that's progress. It may not help predict the end of the project, but I dare say that a manager would like to know that instead of a vague "We're still working on it, but we don't know when will be done". Even if it's a bug you're investigating, a more helpful report would be "I had three hypothesis, I managed to eliminate two" instead of "I'm still on this ticket".
> If 80% of the tests are passing, then another week of work get 2 more passing, that's progress.
Sure. But that is the sort of information you get by ... talking to the responsible dev.
You can get the dev to describe their progress in any one of a myriad of ways. But you aren't escaping from the fundamental issue here of the dev self-reporting and management ultimately having to have confidence in the dev's ability to self-report. There is no more information for non-technical management in this metric than if the dev says "I think I'm about 60% done".
I'd have more respect for a dev that estimated this way than one who just guessed at random, but it isn't making it easier for non-technical people to gauge progress. It is actually running a serious risk of backfiring, because on the date that the metric suggests the project should complete, the dev might quite reasonably discover something that needs another X amount of time to work on something that the test cases didn't cover. Then management may well feel deceived because it turns out the metric wasn't tracking progress accurately.
I agree with you, but it's not that different with any other department. Sales can project 8k sold units for a quarter, but with 3k only being sold after 2 months. It's a more tangible metrics but unexpected things happen and and you just have to trust your subordinates to report them to you so you can react properly.
Managing a software project is not that different than directing a movie or building an house apart from that inspections are easier as the result is visible. But you can make software work visible by having milestones, tasks lists, and tests. Not by following SCRUM methodologies.
That argument is avoiding the point of contention though. It observes that sales can be behind a metric and a dev team likewise. So far so good, but that isn't the problem with the dev metric.
If the sales team lead started the quarter with an 8k goal and ended with 6k actual, they are a quarter closer to being fired. Their job is to hit the metric because the metric is in itself commercially important.
If a software team lead starts the quarter with milestones, tasks lists, and tests, and at the end of the quarter has pushed back a milestone for something unexpected of higher priority, did different tasks because that turned out to be more efficient and refactored the test suite to better reflect quality it isn't at all reasonable to use that as evidence they should be fired. A company that uses those things as a progress measure is going to be bad at software because it can't learn fast enough to pivot effectively. The company might fire the responsible lead anyway under the same logic as they used with the sales team lead, but that would result in them firing their high-performing leads who are more interested in good commercial outcomes than achieving bad metrics.
Some of the highest performing software team leads are happy pivot on a dime mid-quarter and call it a major success even if the initial targets were reasonable. There are no high performing sales team leads who think missing a reasonable target is a win. Acceptable, maybe, but not a win. This is a major problem for non-technical managers and means that milestones, tasks lists, and tests can't be used to track progress.
The only way to improve on that is for managers to go in to git and check what is happening. Technical managers can get deep into tracking progress. Non-technical managers cannot.