I posted this on physics.SE but was directed here. So it is a little specific to that forum. If you've already read all the arguments possible in the original please skip to the last two paragraphs.
The voting system disincentivizes downvoting among low-rep users, who lose almost as much points as the Q/A they're downvoting. This leaves a handful of high-reps who can afford downvoting serial bad answers.
The penalty-sharing ratio is high, for awarding a penalty (or in polite terms, expressing my disapproval) I share 50% of the penalty. Besides, the approval/disapproval ratio is 5 times as high in magnitude. One the one hand, this says a good thing about the site, that the system was created primarily as a positive thing: contribution in any form is encouraged, and higher reputation encourages further participation with privileges. So it is understandable, that one or two bad posts with some downvotes shouldn't be detrimental to that experience. The penalty sharing ratio seems also a way of saying: "Hey new guy, it'd be better to contribute rather than downvote!" Which is well and good.
But this system fails for a (hypothetical) user accustomed to giving serial bad answers. A single deluded yet un-initiated co-conspirator can offset 5 legitimate downvotes by a single upvote. Even some sympathetic and kind-hearted user who thought that the Q\A was bad but not that bad, might try to reduce the downvote count and leave the bad-answerer with a net positive reputation. Give this bad answerer some persistence, as people neglected by formal academia tend to develop if they stick to their field, and this user ranks among the top 15% of users for that forum.
This results in a decrease in quality, which results in some smart people leaving. We all know of one Associate Professor leaving on such concerns.
This issue has been discussed, but I felt it should be posted to express concern after seeing two seniors leave. This is a relatively small community here, and it depends crucially on the (relatively) small number of people who provide quality answers.