In short, what I'm asking for is something like this:
Assume you've got an answer to a question. Now, someone downvotes that, and doesn't leave a comment. You're scratching your head at why, as your answer (you think) was good, answers the question, and works. However, you can't ask the downvoter, as (s)he is anonymous.
My suggestion is that for the first downvoter (and only if there is ONE downvote, as when you have two downvotes it usually means that it's either a bad answer, or it's quite a popular answer and it's just getting voted down for the sake of it) that you can mention him/her in a comment, as
@Downvoter. This would be completely anonymous, and the person would get a notification, but you wouldn't know who it was. So they can choose whether or not to comment (for example, if it was a terribly bad answer, (s)he shouldn't be forced to explain why). However, they would get a notification, so maybe if they forgot to explain why they could be reminded.
Also, I'd say there should be a rep limit for this, as it means that new users can't abuse this if they got downvoted because they posted a bad answer. Not that high, though. Maybe around 250?