It seems like maybe cancelled downvote might be less...weird?
It's not really wrong, but I agree it's a bit weird.
It's even longer so it may not be practical, but I opt for
or as suggested by MDMarra
or as suggested by slhck, even more in keeping with SO's general design
I'm not a fan of the word "cast", so I recommend the following for when you vote up or down on a question:
If you want to replace that with iconography, that works too, but the above is pretty simple (even simpler than upvote/downvote, IMO) and should be understood by anyone looking at it.
However, I don't see the reason to use "removed", since the common terminology when it comes to votes is rescind.
So I'd like to just see "rescind". I don't need to see "upvote/downvote".
IMO, "upvote"/"downvote" is an over-specialization of the voting action. You simply cast a vote up or down, and conversely the action you perform is "rescind", the direction doesn't matter since its reflected in the rep.
You can even take it one step further and say that you can just have "Voted" when a user casts a vote, and let the rep reflect the direction the vote was cast.
If you want to be really consistent you can simply have:
Which ultimately, I like the best, as it reduces four states down to two while being consistent.
Granted, "undownvote" isn't a real word, but expanding it fully would amount to making the vote-type column much wider than it needs to be. "undownvote" keeps it still roughly in the same realm as when there are just normal votes.
How about "withdrawn"? It sounds more of an active verb to me. Or would that be ambiguous as well?