You are thinking of the wrong failure mode. Going too fast to react to a piece of debris is something a well programmed robotic car will never encounter in the first place. It will not go faster than the limits posed by its perception (known) and road conditions (approximately known), and err on the side of caution. It knows no impatience. It is aware of its surroundings at all times, which means it may go faster if it knows it can change lanes, or slower if it perceives a lorry behind it.

In extreme situations where a human would outperform a robotic car, the robotic car will simply be slower. I would, however, assume that in most situations where somebody perceives the car as too slow, what is actually happening is the human overlooking some risk factor.