It seems that to some users, clicking "downvote" would be the way to cancel their upvote. However, if the user has insufficient rep, they receive an error message. Maybe clicking downvote should cancel an upvote (if available) and require a second click to actually downvote (after all, who is likely to go from a positive to a negative opinion?).
Obviously if there is no upvote set, downvote should immediately downvote.
For example, this is the question that sparked me thinking about it.