Have you looked at the heavy downvoters and seen whether the questions/answers they downvote deserve it or not?
Further thoughts: Do the heavy downvoters tend to downvote the newbies? Maybe we should add spawning protection for new users against heavy downvoters.