For some reason, I've observed that people are more naturally inclined to vote on answers rather than questions. To correct this anomaly, the Stack Exchange team put in place some incentives to help the community focus more effort on voting on questions.
Without questions, there would be no answers. Also, oftentimes the quality of the answers is directly proportional to the quality of the question being asked; therefore, to encourage more downvoting on questions, the Stack Exchange team removed the cost to downvoting questions so that the automatic filters would be able to apply question bans on users who post poor questions.
With answers, this really isn't needed, since a single answer has no effect on the quality of the question or the other answers.
Lastly, if an answer is bad enough to where it will have a negative effect on production code, people will downvote it. Oftentimes, just a -1 score is enough to encourage the answerer to fix the post or delete the answer. The cost on answer downvotes is just to help discourage strategic downvoting or overt downvoting on answers that aren't really bad. If answer downvotes were free, we'd likely see a lot more reckless and uncalled-for answer downvoting on what are otherwise helpful posts.