Downvoting an answer already gives the downvoter -1. You don't get your rep back if you leave a comment.
We already suggest that a comment be left with a popup like this for some users:

and this is all that's required. Anything more would likely lead to one of:
- Abuse of the system, like the comments below.
- Downvote sprees against commenters on downvoted answers.
- Less downvoting (and downvoting is useful, even if it's not explained in a comment).
- Duplicate comments when multiple people downvote the post for the same reason.
Requiring a comment could easily lead to comments like:
Your answer missed this important qualification from the question.
I didn't downvote you [Lies!], but your answer missed this important qualification from the question.
This is not a useful answer.
asdfasdfasdfasfasdf
which could be deleted seconds after being posted. The first is useful. The second is useful, but the downvoter felt the need to hide the vote and just plain lied. The third is useless, that's what a downvote means already. The fourth is completely useless. I don't think it's likely that requiring a comment would generate lots of the first type of comment in cases where the user isn't voluntarily posting that sort of comment right now.