This is a very unintuitive feature of the voting system that Jeff assumes makes complete sense.
(Yeah, I don't get it either....)
Basically, after you have voted up or down, even if you remove the vote, you still have technically voted.
I'm guessing this is because a vote might look like this in the Database:
ID | User_ID | Post_ID | Action | Timestamp
2 | 0000000 | 1000000 | 1 | 8/8/93
3 | 0000000 | 1000000 | -1 | 8/8/93
4 | 0000000 | 1000000 | 0 | 8/8/93
Where the first is an upvote, the second is a downvote, and the third is neither a downvote or upvote.
Because when you vote (up or down) you create an entry, it probably doesn't delete that entry when you try to undo it/change it, instead it just sets it to the value you really want.
By not deleting it, you can cut down on the number of queries by doing something akin to:
INSERT INTO database (`User_ID`, `Post_ID`, `Action`, `Timestamp`)
VALUES("$user_id", "$post_id", "$action", "$time")
ON DUPLICATE KEY UPDATE `Action`="$action"
instead of having to try to find if an entry already exists.