Actually, the unlimited downvotes make a clear distinction between "bad" and "worse" answers. And I think everyone will look at an answer and then decide if it has been downvoted enough or not. At least, I tend to avoid downvoting bad answers into oblivion. It still has an educational value to show it's a bad answer.
Actually, bad answers aren't always bad as long as an additional comment explains why it's bad. Thus, if someone else thinks about the same bad idea, they will know it's bad instead of thinking it solves the problem. But things get worse when there are real bad answers that just make the problem worse. For example, someone claims about an unreadable file on his disk and someone else claims it's a virus and the only thing that helps against it is to do a low-level format of the whole disk. Such an advise would be real bad since it results in massive dataloss while the answer is (most very likely) wrong. Such answers need to be downvoted into oblivion to discourage that user from posting more dumb answers.
Do keep in mind that there are users at these SE sites whom are just hunting for reputation. They'll answer almost every question with some answer that they can quickly think of, only to hope someone will give them a vote up. They don't mind a few downvotes since every downvote only lowers their reputation by 2, while each upvote gives them 10 reputation.
Thus, if the number of downvotes were limited, those reputation whores would just be encouraged to post lots and lots of answers, hoping to gain at least one + vote per answer. If the downvotes were limited to 3 max, they could get a net gain of 4 points per useless answer, as long as someone will upvote them at least once. I prefer to downvote these reputation whores into oblivion...
However, I do think that every downvote needs to be accompanied with an additional comment, explaining why it is downvoted. This way, others can just agree with the downvote by upvoting the comment to indicate the answer is bad, without downvoting the answer too much because it isn't real bad.
As suggested by John Smithers, enforcing comments for downvotes would not be very popular, so a better suggestion:
Whenever you downvote an answer, you must leave a comment, but this comment will be anonimous! (Just like the vote.) With upvoting, you should also be able to add an anonimous comment. But for upvotes, this would not be required.
The reason for forced comments for downvotes is that it forces people to explain the downvote and to make it harder for some to do drive-by downvotes in general, since they have to add a comment every time. (And silly comments could then be flagged and deleted by the moderators with the downvote.)
A question with 20 downvotes would then have at least 20 comments and will get noticed. It's likely that it will be flagged sooner or later and if the owner of the answer doesn't delete it, a moderator could probably delete it after evaluating the comments.
