To add a bit more to these answers, Consider the following:
- Click downvote.............Now it's -1.
- Click downvote again...Now it's 0.
- Click upvote.................Now it's +1.
- Click upvote again.......Now it's 0.
Essentially, when you click downvote, and then click the upvote, you are implicitly performing three actions (not just the first two).
The premise here is that it makes more sense to go from -1 to +1 to minimize the number of clicks if you voted in the wrong direction. It might make more sense to change the arrow (up or down) to an X to more clearly signify cancel, but my hunch is that most would rather simply leave the interface unchanged.