Give new edit reviewers some audits to begin with

Question

Apparently many new reviewers approve crap that:

• is just crap 💩 (or spam)
• introduces backticks to emphasize stuff
• add typos, chatspeak and other noise. plz help me kind sirs!

Giving them some audits to begin with would probably help educating them. I'm not talking solely about the Markov-chain spam/vandalism although one of those audits should be like that. There should some perfectly fine edits, some that add backticks and similar crap and some that "correct" a link while changing it into an obvious spam link.

I'm not sure what the action for failed audits at this point should be. Possibly displaying detailed information about why that specific edit was invalid/valid, banning for a short (1 hour maybe) time and then showing another set of review audits. Only after passing the full set of "new reviewer" audits the user would get to review actual edits.

Answer 1

Yes, please. I was about to type up a request for the same.

We have a bit of a problem right now on Stack Overflow, as the Winterbash hat for reviews has caused a bunch of new reviewers to all of a sudden jump into the review queue. Many (most?) of them are just in it for the hat, so they approve everything they see. This is leading to a lot of really terrible or even vandalizing edits being approved at the moment. The traditional audit process isn't working, because you normally have to review a number of items before you hit your first audit. We've been cleaning up the mess from this over the last few days.

I've been observing the review process for a while now, since the audits kicked in, and bad reviewers typically follow the same pattern: they start off approving everything to game the badge (or hat), and then hit their first failed audit or ban. Most people either stop reviewing at that point or reform and actually start paying attention. Only a few continue to abuse the system, and we have ways of picking them out and dealing with them.

The problem is the damage someone can do before they hit that first audit. If you have a bunch of first-time reviewers hitting the system at once, even spam or vandalism can slip through.

Standard audits would be fine (gibberish for suggested edits, etc.), because they seem to have a good track record of catching the worst reviewers. We just need to have the system weighted to present an audit within the first few posts someone sees. This would help to catch bad reviewers much earlier, and reduce the damage they can do.

I could also see using a kind of audit frequency scaling that reduces over the number of reviews you perform without an audit failure.

Answer 2

I agree this is a real issue, I just now saw a bad edit being approved with 2 out of 3 reviewers having never rejected anything (and accepting ~10 edits each so far).

It's fairly easy to see these things happening when you go over the queue, if you click back you can usually see that the edit you just rejected for whatever reason was already approved by these guys. You can do a rollback of course, but they have already finished approving 3 other bad edits by then, it's a lost cause.

What I would like is for there to be some mechanism to mark a bad edit review (or a bad edit for that matter), like a mini flag. I don't want a moderator to immediately take action because a single bad edit/review is not such a crime, but if enough of these mini-flags accumulate (from different users), maybe they should lead to some temporary ban.

You can think of that as a complementary mechanism to audits - you either fail an automatic audit, or fail this new kind of directed audit that some more experienced user decided to run on you after he noticed your actions.