5

A lot of the proposed solutions to bad review audits require extra work, are too ambiguous, don't work in certain review types, or are impossible to implement at scale.

I have a solution I believe should fill those holes reasonably well.

For argument, let's assume:

  • 20% of all review audits are failed
  • The good review audits are failed closer to 10% of the time
  • Bad audits are less common than good audits, but are failed substantially more. Probably close to 50%.

A lot can be gleaned from this information. Namely, audit quality. Over a certain threshold, like 30%, it's probably a bad audit, or at least "bad enough."

I propose removing the post from the audit pool and reversing any failed audits when failure exceeds that threshold (and raw number of reviews is large enough to reduce noise). It might be beneficial to also consider the experience of the reviewers in determining their weight in this calculation.

There may be a SEDE query that could help to justify this feature, but it feels intuitive enough without.

This on its own will do nothing to address complaints, since it's all behind the scenes. What could help address complaints might be changing the Stop! text to mention the automated review of audits, so users are encouraged to forge ahead knowing there may be redemption for their perceived transgression.

Improve this question
6
  • 7
    We don't like to assume things. I've seen plenty of very good audits that have been failed a lot simply because the reviewers actually sucked at reviewing. Just because people are failing it doesn't mean we should stop using it. It could very well be a shining example of an audit that we should be using. You're basically saying that audits should almost always be freshly chosen, which only increases the chances of getting a bad audit.
    – animuson StaffMod
    Mar 25, 2014 at 22:30
  • Would my sentence about considering reviewer experience in the calculation alleviate that concern, or still not quite?
    – mhlester
    Mar 25, 2014 at 22:31
  • 2
    Call it the Stack Exchange Semi-Intelligent Watcher Watcher Watcher™.
    – user102937
    Mar 25, 2014 at 22:31
  • 4
    Considering your entire question revolves around a bunch of made-up percentages, nothing short of actual statistics will alleviate my concerns.
    – animuson StaffMod
    Mar 25, 2014 at 22:32
  • 4
    @animuson: Although, given actual, tweakable stats, the concept does seem sound. We do this already with the StackExchange MegaNuclear QuestionQuality Mashinator™, and the Stack Exchange IsoLinear AskerBanninator™.
    – user102937
    Mar 25, 2014 at 22:35
  • 2
    Given the very large number of posts that are audit candidates, I'd actually be surprised if more than a handful had been used as audits more than one or two times. I don't know that we'd be able to extract any kind of meaningful statistics from individual audit cases.
    – Brad Larson
    Mar 25, 2014 at 23:25

0

Reset to default

You must log in to answer this question.