Either direction is entirely arbitrary. There is no objective basis to claim it's morally better to kill plants instead of animals. These moving lines are being made-up as people go along, and it's almost strictly an issue that exists in the very wealthy first world (ie it's invented by people with nothing better to do, who believe they're superior to the rest of the planet). The best that can be said is: vegans have arbitrarily created a personal moral line, wherein they are ok with killing plants but not animals. Anything else is just subjective argumentation.

And the obvious reason why vegans and others do what they can mentally to avoid facing the plant vs animal hypocrisy, is because if you take that to its logical conclusion, you end up with nutjobs that claim it's as evil to kill an ant as a human baby.