Testing is never enough. In fact, it won't catch 99% of issues by the virtue of them often testing happy paths only, or that they test what humans can think of, and by no means they are exhaustive.
A robust canarying mechanism is the only way you can limit the blast radius.
Set up A/B testing infra at the binary level so you can ship updates selectively and compare their metrics.
Been doing this for more than 10 years now, it's the ONLY way.
I'm not sure that justifies potentially bricking the devices of hundreds(?) of your clients by shipping untested updates to them. Of course it depends... and would require deeper financial analysis.
> They won't be able to test exhaustively every failure mode that could lead to such issues.
That might be acceptable. My point is that if you are incapable of having even absolutely basic automated tests (that would take a few minutes at most) for extremely impactful software like this starting with something more complex seems like a waste of time (clearly the company is run by incompetent people so they'd just mess it up)
A robust canarying mechanism is the only way you can limit the blast radius.
Set up A/B testing infra at the binary level so you can ship updates selectively and compare their metrics.
Been doing this for more than 10 years now, it's the ONLY way.
Testing is not.