The up and down arrows are too small and close together so it's easy to accidentally vote a story or comment the wrong way. This is a problem for me on a phone especially but even on a computer it can lead to an accidental vote.

So without making UI changes, how about allowing the user to change their mind on a vote for a brief amount of time (something like a couple of minutes). This can work by eliminating (or marking as inactive) the arrow they clicked on the first time (similar how you can't downmode replies to your comments).