85

Apparently many new reviewers approve crap that:

  • is just crap 💩 (or spam)
  • introduces backticks to emphasize stuff
  • add typos, chatspeak and other noise. plz help me kind sirs!

Giving them some audits to begin with would probably help educating them. I'm not talking solely about the Markov-chain spam/vandalism although one of those audits should be like that. There should some perfectly fine edits, some that add backticks and similar crap and some that "correct" a link while changing it into an obvious spam link.

I'm not sure what the action for failed audits at this point should be. Possibly displaying detailed information about why that specific edit was invalid/valid, banning for a short (1 hour maybe) time and then showing another set of review audits. Only after passing the full set of "new reviewer" audits the user would get to review actual edits.

  • 16
    More than this, I think we need like a new reviewer training course. Not nessecarly audits as you suggest, but like a whole system of here's what to do, here's an edit, would you approve it, and maybe like for 2-3 minutes, helping to tell what's good/bad. – hichris123 Jan 1 '14 at 17:56
  • 15
    I like this but think it should be education only, no punishment. Treat it more like a "So you are ready to begin reviewing, here's a quick quiz to help you understand how to review" – psubsee2003 Jan 1 '14 at 17:56
  • If they're all audits then there's no point going over it a second time as the new reviewer could just click reject on them knowing what was about to happen. You'd need some "good" audits in there as well so that they have to pay attention. – ben is uǝq backwards Jan 1 '14 at 17:56
  • @hichris123: Yes, that's kind of what I meant – ThiefMaster Jan 1 '14 at 17:57
  • @benisuǝqbackwards: There would be a few sets so the second try has new edits (different order of valid/invalid, different posts) of course. – ThiefMaster Jan 1 '14 at 17:58
  • @psubsee2003 - In my experience, the audits and the warning / short ban you get when you fail one act are surprisingly educational. Lots of bad reviewers get turned around when they fail audits and go on to review reasonably well after that point. – Brad Larson Jan 1 '14 at 18:09
  • @BradLarson I'll certainly will cede to your experience there, I'm not looking at nearly the same level of detailed data as you are. I was hoping that a few sternly worded messages after a couple of failures would be enough, but if it isn't than some sort of punishment might be necessary – psubsee2003 Jan 1 '14 at 18:16
  • @psubsee2003 If they fail that many audits, they're looking at a long review ban. – hichris123 Jan 1 '14 at 18:22
  • @hichris123 if they approve one suggested edit audit, they deserve a long ban. – John Dvorak Jan 1 '14 at 18:57
  • possible duplicate of Take the review tour – juergen d Jan 1 '14 at 19:29
  • 3
    I don't see how this is a duplicate of that other question at all, @juergend. That's saying add a whole new thing (guided tour of how / what reviewing is). This is just modifying the existing review system to start new reviewers with audits, rather than real reviews. – Josh Darnell Jan 1 '14 at 19:48
  • 4
    Yes, please! – Ilmari Karonen Jan 2 '14 at 6:28
  • How do you planning on handling all the bad audit questions that get chosen? Seen plenty of audits where what the audit is looking for is wrong because a bad question was chosen. – Joe W Jan 7 '14 at 18:17
  • 1
    The audit data for this part would not be chosen automatically. I think ~10 sets would be more than enough. Later audits need to be random to prevent people from roboreviewing. But those early audits would help against people who simply don't know what's ok and what's not or people who are too stupid. – ThiefMaster Jan 7 '14 at 21:04
56

Yes, please. I was about to type up a request for the same.

We have a bit of a problem right now on Stack Overflow, as the Winterbash hat for reviews has caused a bunch of new reviewers to all of a sudden jump into the review queue. Many (most?) of them are just in it for the hat, so they approve everything they see. This is leading to a lot of really terrible or even vandalizing edits being approved at the moment. The traditional audit process isn't working, because you normally have to review a number of items before you hit your first audit. We've been cleaning up the mess from this over the last few days.

I've been observing the review process for a while now, since the audits kicked in, and bad reviewers typically follow the same pattern: they start off approving everything to game the badge (or hat), and then hit their first failed audit or ban. Most people either stop reviewing at that point or reform and actually start paying attention. Only a few continue to abuse the system, and we have ways of picking them out and dealing with them.

The problem is the damage someone can do before they hit that first audit. If you have a bunch of first-time reviewers hitting the system at once, even spam or vandalism can slip through.

Standard audits would be fine (gibberish for suggested edits, etc.), because they seem to have a good track record of catching the worst reviewers. We just need to have the system weighted to present an audit within the first few posts someone sees. This would help to catch bad reviewers much earlier, and reduce the damage they can do.

I could also see using a kind of audit frequency scaling that reduces over the number of reviews you perform without an audit failure.

  • 7
    The edit hat and the review hat aren't worth the trouble and make me rethink supporting hats next year on the site where I mod. Hats are fun, but cleaning up some of the mess is not. Also keep in mind that sites without review audits have no formal training or review that is automated. – bmike Jan 1 '14 at 21:37
  • I think it would be a smart idea to factor in the accept/reject ratio of the user into the likelihood that an audit will appear. I see some users who have approved 50+ edits but have only rejected <5. It's possible they've had a remarkable streak of good fortune, but more likely is they are approving crap edits. – Duncan Jones Jan 7 '14 at 13:51
14

I agree this is a real issue, I just now saw a bad edit being approved with 2 out of 3 reviewers having never rejected anything (and accepting ~10 edits each so far).

It's fairly easy to see these things happening when you go over the queue, if you click back you can usually see that the edit you just rejected for whatever reason was already approved by these guys. You can do a rollback of course, but they have already finished approving 3 other bad edits by then, it's a lost cause.

What I would like is for there to be some mechanism to mark a bad edit review (or a bad edit for that matter), like a mini flag. I don't want a moderator to immediately take action because a single bad edit/review is not such a crime, but if enough of these mini-flags accumulate (from different users), maybe they should lead to some temporary ban.

You can think of that as a complementary mechanism to audits - you either fail an automatic audit, or fail this new kind of directed audit that some more experienced user decided to run on you after he noticed your actions.

  • I would recommend you don't roll-back a trivial edit, unless it was wrong. I agree we should be rejecting them, but if they sneak through then I guess a minimal improvement is better than none. – Duncan Jones Jan 7 '14 at 13:48
  • @Duncan, right, edited. I rollback only in cases "bad" means completely wrong, not just minor. – Leeor Jan 7 '14 at 13:55
  • And you're using backticks here for emphasis as part of the joke of suggested edits right? – random Jan 7 '14 at 14:12
  • @random, backticks? where?! OMG, they're creeping up on me everywhere (but seriously, what?) – Leeor Jan 7 '14 at 14:26
  • Read the second bullet in the question – random Jan 7 '14 at 14:28
  • @random, ah, that. Oh boy, fixed that too. Can't have anyone blaming be for jokes around here, that's not professional. – Leeor Jan 7 '14 at 14:35
  • 1
    @Duncan, Random - by the way, here's a fresh example - stackoverflow.com/review/suggested-edits/3748338 – Leeor Jan 7 '14 at 14:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .