It's worth noting that the greater the granularity of your unit, the more likely you will be able to write tests targeted at bugs.
For instance, if you are only testing through the API of the service, you may have a hard-to-impossible time confirming you recover gracefully from certain exceptions. You generally don't have service-level APIs to throw various exceptions intentionally.
Point being, the overzealous testers do have some good points, even if they miss the forest for the trees.
The bug triggered the exception. The test case encapsulates reproducing the bug. The bug is fixed. The bug can no longer be reproduced. As long as the bug remains fixed the test passes..
Another way of thinking about it. Unless your exceptions are a documented part of your API no one cares about them - they only care about the outcome they actually expect. If you construct tests that pass for positive outcomes or fail for any other outcome then your exceptions remain implementation details.
I think GP is referring to nondeterministic exceptions. For instance, if the service under test depends on some other service, then you may need to test the scenario where the other service is unavailable. The exception is not triggered by a bug, it is triggered by an operational externality.
For networking related problems you can deterministically control failures from the test using something like Toxiproxy. This can be especially useful if you’re working out a particular bug (e.g. gracefully handling a split brain situation or something).
A more general approach would be to just run your happy path tests while wrecking the environment (e.g. randomly killing instances, adding latency, dropping packets, whatever).
I’ve found that the latter often uncovers problems that you can use the former to solve.
Testing these sort of things with unit tests can work, but I’m more confident in tests that run on a ‘real’ networking stack, instead of e.g. mocking a socket timeout exception.
Imagine I am implementing a service that queries 20 separate MySQL database servers to generate a report. (I'm not saying this is a good architecture, it's merely to illustrate the point.) I know that sometimes one of the MySQL instances might be down, e.g. due to a hardware failure. When this happens, my service is supposed to return data from the other 19 databases, along with a warning message indicating that the data is incomplete.
I would like to write a test to verify that my code properly handles the case where one of the MySQL instances has experienced a hardware failure. The point is that I can't do this as a strict black-box test where I merely issue calls to my service's public API.
[edit] And of course "testing the externalities" doesn't help here. I can test the MySQL instances and verify that they are all running, but that doesn't remove the need for my code to handle the possibility that at some point one of them goes down.
Second. You've done this/someone else has done this and now you need to maintain it (we've all been there!). In this case my original post holds. Your test suite mocks the databases for unit tests anyway right? So write some test(s) checking that when the various databases are down appropriate responses are given by your service.
Yeah, sometimes for practical reasons you don't want to, or can't test directly across the API, as good testing practice would dictate.
Taken to the extreme, the philosophy I laid out leads to something that looks like "only integration and end-to-end tests" depending on your architecture.
So I try to be pragmatic whenever possible, but I think leaning towards BDD works better, after 18 months of doing it.
For instance, if you are only testing through the API of the service, you may have a hard-to-impossible time confirming you recover gracefully from certain exceptions. You generally don't have service-level APIs to throw various exceptions intentionally.
Point being, the overzealous testers do have some good points, even if they miss the forest for the trees.