You cannot prove that an algorithm does what you want, unless your understanding of what you want is quite formal. But you can prove that an algorithm makes sense and that it doesn't make specific classes of mistake: for example, a median filter has the property that all output pixel values are the value of some input pixel, ensuring that no out of range values are introduced.

Few customers care about proofs. If you can measure how well the method work for the desired task, that is most cases sufficient and in many cases preferred over proofs.

For hobbyists that's enough, for engineers often okay (I find myself in that situation) but for scientists "good enough" means nothing.

Optical metrology relies on accurate equations how a physical object maps to the image plane so in that case analytical solutions are necessary for subpixel accuracy.

I'm worried about how often kids these days discount precise mathematical models for all use cases. Sure, you get there most of the time but ignore foundational math and physics at your own peril.

I don’t discount foundational math. I do a lot of DSP, and many things in audio can be very elegantly solved with math.

The point I was trying to make is that edge detectors and feature descriptors like SIFT and ORB claimed to have a nice mathematical solution when in fact they are just throwing some intuitively helpful math at an ill-defined problem with an unknown underlying probability distribution. For these problems, NNs just perform much better, and the idea that handcrafted feature descriptors have some mathematical foundation is just false. They are mathematical tricks to approximate an unknown model.