I have seen a lot of good answers voted down for no apparent reason (both mine and others). I suspect that many times vote downs are due to lack of background required to understand the answer, however, in most cases voters don't bother to explain why they are voting down, hence they do not give the answerer an opportunity to further explain.
To rectify this, I'd like to suggest splitting the voting down to two types:
Vote down with accountability - requires voter to add an anonymous comment starting with "-1" and explaining why he/she is voting down. Can be performed beby users with 125 reputation and costs 1 reputation point (same as vote downs are now).
Classic vote down - increase required reputation (e.g. 1000 instead of 125) to vote down without accountability and increase cost to reputation (2 instead of 1) and perhaps even require voter to have answered a few questions with vote ups himself/herself (e.g. require Teacher badge).
Also, perhaps block other answerers from down voting and down voters from submitting to other answers to questions without un-down-voting (this can be done automatically when answer is submitted). - Prevent people from trying to make their answer look better in this way.
Edit: I know this is similar to a few other answers (it combines parts of them and adds a bit more).