In my opinion, the current downvote system has two big problems.
Firstly, you can't remove a downvote without adding an upvote, effectively adding two to the post's score. Perhaps I think that my downvote was unwarranted but I'm certainly not considering an upvote.
The second problem is that on a recent post, I couldn't change my downvote until the user edits. Why is this? Perhaps a comment, for example, informed me of why this post didn't deserve a downvote. This doesn't mean that the answer should have to be changed to merit changing my downvote.
Ultimately, I feel like my votes are my votes, and the site has no right to tell me how and when I can and cannot apply or change my votes, provided that I'm not being abusive with them, but I feel that it's currently restricting me from expressing my opinion, especially in the scenario where my initial opinion is changed.