It is not obvious that clicking the same arrow that you just clicked to up-vote or down-vote will have the effect of undoing that action. Not obvious at all. Intuitively, one might think that clicking the same arrow again would either (a) vote again, same direction or (b) more likely, be a no-op or result in an error message: "Ignored - only one vote, please".
What's more, it is pretty natural to think that if you just added one to the up votes then clicking the down arrow would cancel that action, and vice versa. If you try that you find that it instead both (a) cancels your up vote and (b) adds a down vote.
Not obvious/intuitive at all. That is, not very guessable. But once you know about it, it is of course a reasonable UI.
All that's needed is to make the behavior more obvious. Provide some visual clues. Perhaps even pop up a hint when someone switches arrow direction, e.g., votes up then down, suggesting, say, "To cancel your vote, click the SAME arrow again".
Just look at the number of times people have asked how to cancel their vote -- search for
[voting] cancel. It should be clear that how to cancel a vote is not being communicated well enough.