Today I voted up the wrong answer. I noticed that there's no way to undo your votes. When you have voted an answer you are forced to vote it, even if you don't want to. I immediately noticed the error and I could just change my vote to up or down, not delete my vote. It makes no sense at all, imho. Since you are able to change your vote within 5 minutes, you should also be able to remove your vote entirely. I can't believe no one noticed that before...
Click on the upvote icon (the up arrow) to upvote.
Click again to undo your upvote.
Click on the number to display the upvote and downvote counts, instead of just the total score. (Note that you must be an established user to do this.)
You can cancel it: click on the vote you've cast again, i.e. click on the up arrow again to cancel an upvote.
As you've noticed with up / down you can only do this for five minutes. I did exactly the same as you and ended up downvoting to cancel at first. At least I got the critic badge out of it :-)