23

So now we have review audits on all the queues. Is anyone measuring anything (even the rate of whiny questions on meta about robo approvers) to see if it's making a difference? Some possible numbers:

  • absolute numbers of reviews per day: I would expect these to drop as the grinders are either "on a break" or giving up on it)
  • average time spent per review: I would expect this to increase for everyone, or because the sub-one-second grinders are not contributing as many reviews to the pot)
  • churn in the reversible queues (eg close/reopen): I would expect this to drop when less bad decisions are being made
  • edit rollbacks, especially if there's a suggested edit in the history: I would expect this to drop

Do we have access to any of that? Can we be told how many people are "on a a break"? The relative proportions (eg 50% of all suspensions are in the First Posts queue?) The most robo-y queue?

Any numbers at all to make us feel like the queues are being made into happier places?

Improve this question
8
  • 1
    Sub one second reviews are not possible (I'm not sure if they ever were) because the buttons are disabled for a few seconds.
    – user200500
    Feb 12, 2013 at 22:21
  • still it's a rare non-grinder who can click those buttons the moment they appear. Most of us take longer to read the edit than the button delay. So removing grinders should raise the average time spent per review. Further, nervousness that you might fail an audit could be expected to make some non-grinders read a little more carefully than they were before, just to be sure.
    – Kate Gregory
    Feb 12, 2013 at 22:43
  • True enough; average review time would be an interesting stat. If it is very near the button cooldown we know there`s a problem.
    – user200500
    Feb 12, 2013 at 22:48
  • 5
    I'm not sure how many of the internal statistics I can share publicly, but yes, the review audits seem to have had a huge positive effect. By banning a small number of very bad robo-reviewers, we all but eliminated the upvotes that I would see daily on spam and non-answers. Early indications are that suggested edit review bans are reducing the number of really bad approvals. Again, I'm not sure what numbers I can share, but that's my qualitative take on this.
    – Brad Larson
    Feb 12, 2013 at 22:53
  • 2
    Shog9 had this to say when the audits started in December: "Quick stats: 348 users have failed at least one audit, 117 two, 36 four. That's out of 1471 reviewers who have been audited at least once. At four failures, I start looking closely - four in a week, very closely."
    – Brad Larson
    Feb 12, 2013 at 22:54
  • 3
    imo, combine those last two into comments into an answer
    – user133440
    Feb 12, 2013 at 22:59
  • 2
    Anecdote: I almost mod messaged someone on Programmers yesterday for a horrible edit before I realized it was an audit.
    – yannis
    Feb 13, 2013 at 6:11
  • "rate of whiny questions on meta" is pretty easy to measure: these posts are (should be) tagged review-abuse and listed to anyone in a convenient way when sorted by newest
    – gnat
    Feb 13, 2013 at 7:01

1 Answer 1

Reset to default
15

It's been a little while, and we now have access to some more fine-grained statistics, so I can flesh out the comment I made above. Yes, the review audits seem to have had a significant positive impact on the quality of reviews, at least when it comes to suggested edits.

When the suggested edit audits first launched, the percentage of failed audits was over 10%. With the exception of the occasional good reviewer who clicks the "edit" option on those, there is no way any of these should be approved. They are complete gibberish. Over the last month, 10% of these audits have been failed, over the last 14 days 8%, and only 7% in the last week.

Fewer reviewers are approving these vandalizing edits, due to review bans and they way they've seemed to educate the worst reviewers. I've watched, and many of the reviewers that were clearly spamming approvals on everything have since become a lot more critical of the things they are reviewing. Rarely have I seen someone come back after being banned from review for a few days and pick up right were they left off.

With the other queues, we've seen the robo-upvoting of everything in the Late Answers or First Posts queues all but disappear. Spam and non-answers are no longer being voted up, but instead are being properly flagged. I've actually been going back and cleaning up some of the mess that was left behind from that, and it's striking just how much spam slipped through in the period from when review badges were introduced to the point were audits kicked in.

I still think there are a lot of ways these audits could be improved, from the removal of some of the more confusing deleted answers used as audits, to better handling of the way that people try to improve edit audits, but overall they seem to have been a success.

Improve this answer
1
  • 2
    That is good news. Keep up the good work.
    – Toon Krijthe
    Mar 16, 2013 at 20:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .