14

Audits are a terrible metric. They have little significance within the scope of reviews, and say very little about the reviewer. Using an algorithm to select audits doesn't work particularly well, and there will always be exceptions which aren't covered by the algorithm. This means that the value of auditing is limited, at best.

Anyone looking through a user's history will simply look through their entire history; audits serve no purpose other than to instate bans. Specifically, for each audit type, we see these complaints with significant frequency:

  • Close Vote audits are often incorrect, both because of old questions and because people sometimes miss things
  • Suggested Edit audits are blatantly obvious (more so than the others), and anyone with eyes would see it (most of the time). This makes SE audits hard to fail, and doesn't actually gather significant data about the quality of reviewing.
  • First Posts audits are very easy to detect, and also very easy to game. Several methods for doing so have been pointed out in the past, though I'm not going to list them here.

Each of the audit types has its flaws, each can be easily bypassed, all of them have frequent errors, and as a result, none of them truly distinguish the quality of a reviewer. There have been many pushes on Meta in the past to see audit statistics, as well as questioning the effectiveness of the audit system. Piles are designated towards pointing out bad audits. Take a look at the related questions list to the right of this question.


Proposed is an entirely new system for tracking reviews and reviewers' effectiveness.

Instead of maintaining an audits system which can easily be gamed and hacked around, simply audit based on quorum. This would work in a similar way to the banning system; once you reach a low enough internal score, you're suspended for two days, and given a link to your review history.

  • If a user picks an option on an audit which is different than the decided action, then their audit score decreases.
  • If a user matches the quorum for an audit, then bump their audit score up a bit.
  • If that action leans more towards doing nothing (e.g. "No Action Needed"), penalize the user significantly more than if they took action.
  • More contentious reviews would be weighed less (e.g. 3 accepts, 2 rejects).
  • Less contentious reviews would be weighed more (e.g. 1 accept, 4 rejects).

This sets up a system where, if the user takes no action continuously, they're suspended for two days. If they, for instance, reject edits continuously, they're also going to be suspended and asked to review their history. However, if a user has a couple discrepancies compared to what was decided, nothing serious will happen, and the instances will be overlooked.

Properly tuned formulas for weighing the significance of their review would negate any suspensions which would arise from a few differing opinions. However, if a user's opinion is consistently different than the majority, then they may be suspended - they should look to see what they're doing wrong. The formulas would be very easy to tweak, since calibration could be done in the background while review audits are still active. The formulas could be perfected before they were ever implemented.

This would need to go hand-in-hand with requiring more reviewers for First Posts.

What are your thoughts? What problems do you foresee with this system, and how would you change it to fix them?


To address issues concerning immediate feedback, here are several options:

  • First, we could keep selected audits for users under 250-300 reputation.
  • We could notify users when their reviews consistently don't match quorum
  • We could show users an introduction page to each review, similar to the "How to Ask" page. Some argue that this page would be totally ineffective, however, its purpose would be to let the user know that they're being held to standards. If they're reviewing posts, they're aware of this fact to begin with.

Since we should probably be finding ways to give users better review feedback, this is an opportunity to do so.

To address issues concerning the self-perpetuation cycle of the review queues, and the possibility that bad reviewers will trump good ones, here are my thoughts:

I don't this will happen, actually. I have confidence that most of the reviewers out there will take some form of appropriate action. Still, it's a valid concern, so:

A possible solution is to give moderators significantly more weight than regular users, and ask them to spend a few days focusing down the review queue. I think that, once the status quo has changed, it will stay that way by its own self-perpetuating cycle. Moderators are there to help guide users, so if it needs a small push every now and again, a moderator can step into the queue.


These are several ideas, and are issues which would need to be addressed, so please post your suggestions below.

  • 11
    Heh, I was considering this just a few hours ago. My major concern would be that people who are actually paying attention end up in the minority in more cases than not (especially as far as suggested edits go, even though they're a bit special), which would undesirably result in the system working against them. – Tim Stone Jul 11 '13 at 22:11
  • @Tim While yes, that is potentially a side effect, I don't think that would happen. I have confidence that most of the reviewers out there will take some form of appropriate action. Still, it's a valid concern. – Aza Jul 11 '13 at 22:12
  • 3
    I like it. The score algorithm would have to be tweaked until it successfully identified the offenders with minimal false positives, but I don't see why this wouldn't work. – Robert Harvey Jul 11 '13 at 22:14
  • @Robert That would be very easy to do. The formula could be tweaked in the background before it ever went live. – Aza Jul 11 '13 at 22:16
  • @Tim Another possible solution to that would to give moderators significantly more weight than regular users, and ask them to spend a few days focusing down the review queue. I think that, once the status quo has changed, it will stay that way. If it needs a push, then well, that's what mods are for. – Aza Jul 11 '13 at 22:22
  • I believe I recall review of suggested edits by majority rule being proposed before the review queues and audits existed. I'll see if I can find it... – Josh Caswell Jul 11 '13 at 22:29
  • @Josh The first one is a similar question, but it uses a rather faulty metric of simply "count the majority and minority"; that's different from this, which would use an overall weight. – Aza Jul 12 '13 at 1:08
  • 3
    I'm all for this except for suggested edits. Too often bad edits are approved. I don't want a majority of bad edit reviewers killing off the good edit reviewers because the only way this idea will work is if the majority is correct. – psubsee2003 Jul 12 '13 at 1:46
  • @psub See my comment above; once the status quo has been set, it will take a lot of force to change. Give moderators more weight, and have them hang out in /review for a few days, and the status quo will shift. – Aza Jul 12 '13 at 1:47
  • @EmrakultheAeonsTorn let me reread your post, maybe I am missing something – psubsee2003 Jul 12 '13 at 1:49
  • @psub It's this comment; sorry about the confusion. I'll add that to the OP. – Aza Jul 12 '13 at 1:49
  • @EmrakultheAeonsTorn no, I saw that... maybe I am confused by the nature of the post that is used for the "audit". Are you proposing using real reviews and scoring based on how others act on the same post. Or are you still using audit questions and scoring based on how everyone else felt about the same audit question? – psubsee2003 Jul 12 '13 at 1:52
  • @psub This would replace the audit system entirely; it would be self-sustaining, and would weigh your actions against the status quo. – Aza Jul 12 '13 at 1:54
  • @EmrakultheAeonsTorn yeah, I saw that... I understand the words, I guess I'm just not getting the meaning. Either it is very late and my brain is ready for bed, or your way over my head here. – psubsee2003 Jul 12 '13 at 2:01
10

I've been thinking about it for some time, and there are some holes.

The biggest problem I can think of is that the current system gives immediate feedback. This is very important, especially when one is just getting started reviewing. I failed some review audits right up front, and learned because of it.

After this problem is taken care of (and the kinks are ironed out), though, it could be a great system.

  • While yes, this could be an issue, I think there are better ways of teaching people how to review than by having them fail reviews; that's one of my current complaints. Since this system wouldn't be quite like the question banning system, it would be possible to warn users when they're not following conventions consistently; that would give them a chance to review their actions before they've been suspended. – Aza Jul 11 '13 at 22:34
  • 1
    @EmrakultheAeonsTorn True, but currently it's important for it to be immediate. I'm open to feedback, though. – Undo Jul 11 '13 at 22:35
  • I have separate thoughts on how we should be initializing new reviewers, and including them in this post would make it too broad. Maybe chat would work, though. – Aza Jul 11 '13 at 22:36
  • For public information: The link to our chat record is here. – Aza Jul 11 '13 at 23:50
0

At least for the first-posts and late-answers queue, you can use something other than other reviews for detecting bad reviews, namely the future fate of the post, for example:

  • Reviewer selects no action needed or votes up; post gets closed, deleted or heavily downvoted afterwards → increase audit score
  • Reviewer selects no action needed or votes up; post gets deleted as spam or offensive → heavily increase audit score
  • Reviewer votes down, post receives a lot of upvotes per views → increase audit score

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .