4

I have noticed a change in the review audits, and from others feedback, I was wondering if these audits are effective.

I failed this review audit. I was going to flag it, as I could not find the question, just a long discussion, offering conjecture. The OP, certainly seems to have a lot of knowledge, but I suspected it was provoking a polled and opinionated discussion.

See the first sentence of this answer

My final answer consists of two thesis:

This implies there is not a clear cut answer to this question. If a user has to commence an answer with these are my theories.. what does this say of the question being a good fit?

Am I missing something, or is this not the best sample to use for a review audit?

Who chooses questions for review audits/or are they automatically chosen?

How are the review audits monitored?

I would like a discussion on this with a view to improving the audits. These audits are not necessarily going to trip up the reviewers they are designed to trip up (maybe I am a bad reviewer, you be the judge), but I am finding that some users who, also, may fail these, are thinking a lot about these posts and the reasoning behind there actions may not be altogether poor.

Is this type of auditing going to lead some reviewers from getting banned without good reason?

Improve this question
1
  • For what it's worth, I agree with the sentiment here. I failed an audit for a question that I thought was pretty subjective and should maybe be moved from SO to Programmers. The ironic thing about the audit question is it was the question I debated internally more than any others, not an obvious "are you paying attention" question. Using an algorithm to select audit questions is surely effective, but I'm not sure that it is an especially effective teaching tool.
    – Andrew
    Jan 13, 2014 at 4:19

1 Answer 1

Reset to default
5

No, it's not going to lead to anyone that pays consistent attention being given some time off from reviewing. Audits are random, and present you with one of two things:

  • Stuff that we determine to be good content, from an algorithm
  • Stuff that we determine to be bad content, from a different algorithm

And by algorithm, I mean it in the simplest sense. Content that has been on the site for a while, has more than a trivial amount of views, upvotes and upvoted answers is extremely likely to be content that we consider to be good. Likewise, content that was removed due to validated quality related flags is very likely to be the sort of stuff that we don't want around.

All these audits do is check to see if you're paying attention, because 99% out of the time when you're presented with what is 'known' good or bad content, you'll take a predictable action. These tests do not take into account that sometimes people find themselves at odds with how the content was classified.

You'll be presented with deleted posts that you probably feel should not have been deleted, or what the system thinks is good quality that you simply see no value in. That's expected. In order to get some time off, you need to miss these consistently in a manner that shows you were really just going through the motions to get a badge. While the tests do help hone your moderation skills, they're really just in place to catch people who 'rubber stamp' their way through the queue without really paying attention.

As long as you're thinking, you shouldn't have anything to worry about. If that's the case then 'failed' basically just means 'I disagree with how this post was treated' - and you can just move on. If that happens all the time, then you might wonder if you understand what we consider to be good or bad for a site when it comes to quality.

Improve this answer
2
  • 1
    ty Tim, I think that is a really good answer. And yes failing all the time would show that a users notion of what the site requires and what is actually required is at odds. I have failed two audits in a row and felt puzzled. And, of course, do question that maybe my thinking is wrong. Generally one would think that the algorithms the site uses should select a range of posts that would not always be open to query as this one (I believe) is.
    – user310756
    Jul 16, 2013 at 16:21
  • More accurately, "If that happens all the time, then you might wonder if you understand what our audit selection algorithms consider to be good or bad for a site when it comes to quality."
    – Jason C
    Aug 17, 2013 at 3:24

You must log in to answer this question.