Honestly that just says that the interface is too low level. Telling a car to drive you to some place and make it fast is how we interact with taxi drivers. It works fine as a concept, it just needs a higher level of abstraction that isn't there yet.
It's easier up until it's time to drop you off, and the selected dropoff point is suboptimal or plain impossible to stop at, and you want to give the car last-minute directions. Then the traditional, "human driver way" of looking out the window and telling them where to go based on what you see is far superior than trying to perspective-switch between the 3D situated view and imprecise, finicky 2D map interface.
A perfect interface would be a combination of both ways. Also it depends on personal preferences. I learned to use a paper map as a teenager and it’s convenient for me but I know some people struggle to find a way even using a map on a smartphone.
This only works for tasks where the details of execution are not important. Driving fits that category well, but many other tasks we're throwing at AI don't.
