87

Apparently many new reviewers approve crap that:

  • is just crap 💩 (or spam)
  • introduces backticks to emphasize stuff
  • add typos, chatspeak and other noise. plz help me kind sirs!

Giving them some audits to begin with would probably help educating them. I'm not talking solely about the Markov-chain spam/vandalism although one of those audits should be like that. There should some perfectly fine edits, some that add backticks and similar crap and some that "correct" a link while changing it into an obvious spam link.

I'm not sure what the action for failed audits at this point should be. Possibly displaying detailed information about why that specific edit was invalid/valid, banning for a short (1 hour maybe) time and then showing another set of review audits. Only after passing the full set of "new reviewer" audits the user would get to review actual edits.

14
  • 17
    More than this, I think we need like a new reviewer training course. Not nessecarly audits as you suggest, but like a whole system of here's what to do, here's an edit, would you approve it, and maybe like for 2-3 minutes, helping to tell what's good/bad.
    – hichris123
    Jan 1, 2014 at 17:56
  • 15
    I like this but think it should be education only, no punishment. Treat it more like a "So you are ready to begin reviewing, here's a quick quiz to help you understand how to review" Jan 1, 2014 at 17:56
  • If they're all audits then there's no point going over it a second time as the new reviewer could just click reject on them knowing what was about to happen. You'd need some "good" audits in there as well so that they have to pay attention. Jan 1, 2014 at 17:56
  • @hichris123: Yes, that's kind of what I meant Jan 1, 2014 at 17:57
  • @benisuǝqbackwards: There would be a few sets so the second try has new edits (different order of valid/invalid, different posts) of course. Jan 1, 2014 at 17:58
  • @psubsee2003 - In my experience, the audits and the warning / short ban you get when you fail one act are surprisingly educational. Lots of bad reviewers get turned around when they fail audits and go on to review reasonably well after that point. Jan 1, 2014 at 18:09
  • @BradLarson I'll certainly will cede to your experience there, I'm not looking at nearly the same level of detailed data as you are. I was hoping that a few sternly worded messages after a couple of failures would be enough, but if it isn't than some sort of punishment might be necessary Jan 1, 2014 at 18:16
  • @psubsee2003 If they fail that many audits, they're looking at a long review ban.
    – hichris123
    Jan 1, 2014 at 18:22
  • @hichris123 if they approve one suggested edit audit, they deserve a long ban. Jan 1, 2014 at 18:57
  • possible duplicate of Take the review tour
    – juergen d
    Jan 1, 2014 at 19:29
  • 3
    I don't see how this is a duplicate of that other question at all, @juergend. That's saying add a whole new thing (guided tour of how / what reviewing is). This is just modifying the existing review system to start new reviewers with audits, rather than real reviews. Jan 1, 2014 at 19:48
  • 4
    Yes, please! Jan 2, 2014 at 6:28
  • How do you planning on handling all the bad audit questions that get chosen? Seen plenty of audits where what the audit is looking for is wrong because a bad question was chosen.
    – Joe W
    Jan 7, 2014 at 18:17
  • 1
    The audit data for this part would not be chosen automatically. I think ~10 sets would be more than enough. Later audits need to be random to prevent people from roboreviewing. But those early audits would help against people who simply don't know what's ok and what's not or people who are too stupid. Jan 7, 2014 at 21:04

2 Answers 2

56

Yes, please. I was about to type up a request for the same.

We have a bit of a problem right now on Stack Overflow, as the Winterbash hat for reviews has caused a bunch of new reviewers to all of a sudden jump into the review queue. Many (most?) of them are just in it for the hat, so they approve everything they see. This is leading to a lot of really terrible or even vandalizing edits being approved at the moment. The traditional audit process isn't working, because you normally have to review a number of items before you hit your first audit. We've been cleaning up the mess from this over the last few days.

I've been observing the review process for a while now, since the audits kicked in, and bad reviewers typically follow the same pattern: they start off approving everything to game the badge (or hat), and then hit their first failed audit or ban. Most people either stop reviewing at that point or reform and actually start paying attention. Only a few continue to abuse the system, and we have ways of picking them out and dealing with them.

The problem is the damage someone can do before they hit that first audit. If you have a bunch of first-time reviewers hitting the system at once, even spam or vandalism can slip through.

Standard audits would be fine (gibberish for suggested edits, etc.), because they seem to have a good track record of catching the worst reviewers. We just need to have the system weighted to present an audit within the first few posts someone sees. This would help to catch bad reviewers much earlier, and reduce the damage they can do.

I could also see using a kind of audit frequency scaling that reduces over the number of reviews you perform without an audit failure.

2
  • 7
    The edit hat and the review hat aren't worth the trouble and make me rethink supporting hats next year on the site where I mod. Hats are fun, but cleaning up some of the mess is not. Also keep in mind that sites without review audits have no formal training or review that is automated.
    – bmike
    Jan 1, 2014 at 21:37
  • I think it would be a smart idea to factor in the accept/reject ratio of the user into the likelihood that an audit will appear. I see some users who have approved 50+ edits but have only rejected <5. It's possible they've had a remarkable streak of good fortune, but more likely is they are approving crap edits. Jan 7, 2014 at 13:51
14

I agree this is a real issue, I just now saw a bad edit being approved with 2 out of 3 reviewers having never rejected anything (and accepting ~10 edits each so far).

It's fairly easy to see these things happening when you go over the queue, if you click back you can usually see that the edit you just rejected for whatever reason was already approved by these guys. You can do a rollback of course, but they have already finished approving 3 other bad edits by then, it's a lost cause.

What I would like is for there to be some mechanism to mark a bad edit review (or a bad edit for that matter), like a mini flag. I don't want a moderator to immediately take action because a single bad edit/review is not such a crime, but if enough of these mini-flags accumulate (from different users), maybe they should lead to some temporary ban.

You can think of that as a complementary mechanism to audits - you either fail an automatic audit, or fail this new kind of directed audit that some more experienced user decided to run on you after he noticed your actions.

7
  • I would recommend you don't roll-back a trivial edit, unless it was wrong. I agree we should be rejecting them, but if they sneak through then I guess a minimal improvement is better than none. Jan 7, 2014 at 13:48
  • @Duncan, right, edited. I rollback only in cases "bad" means completely wrong, not just minor.
    – Leeor
    Jan 7, 2014 at 13:55
  • And you're using backticks here for emphasis as part of the joke of suggested edits right?
    – random
    Jan 7, 2014 at 14:12
  • @random, backticks? where?! OMG, they're creeping up on me everywhere (but seriously, what?)
    – Leeor
    Jan 7, 2014 at 14:26
  • Read the second bullet in the question
    – random
    Jan 7, 2014 at 14:28
  • @random, ah, that. Oh boy, fixed that too. Can't have anyone blaming be for jokes around here, that's not professional.
    – Leeor
    Jan 7, 2014 at 14:35
  • 1
    @Duncan, Random - by the way, here's a fresh example - stackoverflow.com/review/suggested-edits/3748338
    – Leeor
    Jan 7, 2014 at 14:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .