Well, when you get enough reputation and get engaged enough in a site - you're probably not going to want to downvote things for fun. I often find the same folks hanging around tags they care about, even on smaller sites. Downvoting is actually an essential part of content curation on regular sites, and people shouldn't take it lightly.
I think we make a few assumptions in the question that are incorrect. Voting is anonymous and there's no way to figure out who voted on a specific question
As for answers - I don't always answer stuff on everything I know about, and on many sites, tag badges are difficult to get. You're essentially expecting users to get a minimum of 100 rep spread over many answers, for something that is semi trivial to get now by design on each tag there is.
minimum appropriate reputations in all tags of the question/answer
Just makes downvoting impossible, and encourages the use of tags tactically. It kind of just isn't going to work at all, unless your goal was zero downvotes. Oh and stealing from the comments - (sorry @fbueckert) - no one could ever downvote a question with a newish tag until someone got an appropriate badge on it.
"Subject matter" should be given appropriate higher weightage over clarity, spelling, readability, etc!
To an extent, quality issues are fixable. Unless of course, its hard to tell what the post is about. In a sense, we expect a minimum effort for comprehensibility and that gives people a better chance to understand if your post reflects your knowledge of the subject matter.