13

The robo-reviewer issue seems to be rampant on Stack Overflow, and I was wondering if audit failures really stop robo-reviewers.

I'm looking for information as to whether the audits are effective at stopping robo-reviewers from their robotic reviews (possibly looking at the approved-to-rejected ratio, number of audits failed, etc.).

After failed audits, do robo-reviewers really shape up and start making good audits? (Needless to say, if they don't shape up, then this has some huge ramifications on the whole audit system.)

  • 2
    They are temporarily banned, in the most egregious cases, and the length of those bans increases if the reviewer persists. Banned users can't make bad reviews. – Robert Harvey Apr 7 '14 at 22:38
17

Your question is very close to the one I answered here, so I can point you at that for some cobbled-together stats from soon after the audits were put into place.

Since then, we've had a lot more time to observe the review queues and how audits have impacted them. In my opinion, the audits have proven to be absolutely vital in maintaining quality within the review queues.

Almost every reviewer who tries to game the system for a badge hits the audits and either stops reviewing or reforms. When the audits were first put into place on the suggested edit review queue, it was stunning to watch reviewers who hit "approve" on everything get banned and come back as good reviewers. This pattern carried over to the other review queues.

We've seen what happens when you have a review system without audits. We're still finding spam that was not just approved but upvoted multiple times during the month or so between badges being added and the audits coming on line. I can almost guarantee that the site would go right back to that if the audits were removed.

There are only a small handful of people who keep trying to abuse the review system after being temporarily banned due to audit failures. Moderators have tools to suss out and deal with these bad reviewers manually, but there are only a relatively small number of them out there.

Sure, some aspects of the audits can be improved, and it would be nice if the system could catch a few more of those that the audits don't. However, I'm actually surprised that this technological solution to a social problem has worked as well as it has.

  • That's perfect. Thank you for the insight! – Blue Ice Apr 8 '14 at 2:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .