139

Some people are complaining about the automatically generated audits. They are based on close reasons and apparently are a little controversial :-)

I propose that for "good reviewers" (definition TBD) an extra UI component be added to let the reviewer suggest this as an audit item. Possibly:

  • Reject and add as audit button (Close and add as audit, etc)
  • checkbox next to Reject, Close, etc
  • checkbox on the Reject reason, Close reason etc dialog (could get messy, and some actions such as Reopen don't involve a subsequent dialog)
  • dialog after rejecting, closing, Recommend Deletion, etc

Further, after passing or failing an audit, we could ask "good reviewers" "was this a good audit"? Bad audits, whether auto-generated or human suggested, could be removed from use if enough "good reviewers" identified them as such.

Now, what's a good reviewer? I don't want to judge only on behaviour in the review queue. I think flagging well is a good sign. So is editing - maybe an edit-related badge? And yes, some number of reviews would be good. Maybe some ratio of pass to fail on review audits. But not JUST what we do in the queues.

(I first suggested this in my answer to https://meta.stackexchange.com/a/159573/147247 but I'm bringing it up as a feature request of its own and to apply to all the queues.)

Improve this question
19
  • 3
    Hmm, if we can't have decent audits to trap bad reviewers, then how are we going to come up with a system to detect good ones? ;)
    – Bart
    Commented Feb 20, 2013 at 19:25
  • 4
    @Bart - that's why behaviour outside the queues is key. Someone with 1000 helpful flags (or probably with any), with Copy Editor, with Yearling, etc is probably someone who is a "good reviewer" if you add some review accomplishments to the list. Someone with Steward in one queue and nothing else - probably grinding the remaining queues for more Stewards
    – Kate Gregory
    Commented Feb 20, 2013 at 19:28
  • 10
    Agreed. Or why not take this further and have good reviewers review reviews, instead of audits?
    – Pekka
    Commented Feb 20, 2013 at 19:43
  • 37
    Then we'll need review reviewer reviewers reviewing review reviewers, @Pekka.
    – Shog9
    Commented Feb 20, 2013 at 20:08
  • 10
    Quis custodiet ipsos custodes?
    – David Robinson
    Commented Feb 20, 2013 at 20:39
  • 17
    @Shog why? That's a tired argument and I don't buy it. What are the actual chances of 3 site members in good standing wrongly deciding that somebody else's bad review is good, or good review is bad? And even if that happens once, what are the chances of it happening a second time to the same person? Reviewing reviews would also lead to every, or at least many review actions being judged, as opposed to occasional DUI checks as the audit system does.
    – Pekka
    Commented Feb 20, 2013 at 21:35
  • 4
    @Pekka웃 agree - especially if we give no incentive (especially not "ok then you didn't fail after all") for reviewing an audit, and we offer the capability only to "good reviewers"
    – Kate Gregory
    Commented Feb 20, 2013 at 21:51
  • 7
    Really, @0A0D? Nothing about this looks a bit... destructive to you?
    – Shog9
    Commented Feb 21, 2013 at 23:07
  • 3
    You know how tricky the datasources car part can be @Shog9...
    – Kate Gregory
    Commented Feb 21, 2013 at 23:14
  • 2
    @Pekka: in all seriousness, I think there's a place for that - but routinely subjecting all reviews to meta-reviews would be a ton of overhead. A more practical model is one that uses some combination of user input (flags, editing, voting) and heuristics to identify areas that need further review.
    – Shog9
    Commented Feb 22, 2013 at 3:38
  • 3
    I would love to get this one added to the list :)
    – Jack
    Commented Feb 22, 2013 at 10:32
  • 4
    I don't agree with the closure of this question as a duplicate of Review audits and “I understand” button. True, both questions propose a similar mechanism to provide feedback about whether an existing audit is good/bad. However this question also proposes a mechanism (which involves regular users) to create new audits...
    – Old Checkmark
    Commented Jul 22, 2013 at 12:02
  • 1
    ... While the proposed mechanism to create new audits is similar to the proposed mechanism to review existing audits, the goal is opposite and as such has different advantages and drawbacks. I think this would mean that the 2 mechanisms could be considered somewhat separately. An analogy would be: "I want to use tool A to do B and C" (this question) , the other question is about "I want to use tool A to do C".
    – Old Checkmark
    Commented Jul 22, 2013 at 12:07
  • 7
    Did you know you get no notification if your question is closed? Thanks @doubleDown for commenting. I don't see "let humans create new audit entries" as a dupe of "let humans say they didn't like the code-chosen audit entry they were shown" at all.
    – Kate Gregory
    Commented Jul 22, 2013 at 12:46
  • 3
    The problem is, that the criteria for off-topic questions have changed and many 'quasi-good' questions from past are terrible questions now. Many community members are still thinking with old criteria. Unter those circumstances blindly chosing autits basing on upvotes is simply not a good idea.
    – Cjxcz Odjcayrwl
    Commented Nov 29, 2013 at 7:45

1 Answer 1

Reset to default
65

While there is no "officially implemented" solution for this, one can use whatever means are at their disposal now in order to bring the "human factor" to audits selection.

When you spot a slippery audit, go straight to the "item" it uses and do the action opposite to audit direction.

  • If you feel something rotten before submitting the audit, use link to the audit item to get there for corrective action. If you found that you were screwed after the audit, bad audit and item can be found in the activity tab in reviews subsection.

If the audit item has been wrongly served as "known good", down / close vote it. If it was pretending to be "known bad" against your judgement, vote it up / reopen.

I always do this to audits I disagree with.

As I typically open the items in queue in separate tab (for more thorough review), it often happens that I spot slippery audit and perform "correction" even before completing review. It feels somewhat weird to click Looks OK at the item you just downvoted but oh well. I am not going to decrease my "audit weight" just because of a mistake in automatic selection algorithm, and knowing that reviewers after me won't get into this trouble anymore makes it less painful.

Improve this answer
12
  • 17
    This does nothing to help when you are banned from the review queue for 'failing' audits despite having less than 1% disputed or rejected reviews.
    – nathanchere
    Commented Oct 13, 2014 at 7:41
  • 8
    @FerretallicA agree, my approach is only a workaround that merely saves next reviewer from getting unfair suspension. Real solution would be to either introduce human verification of reviews as suggested here or allow challenging and invalidation of wrong ones as suggested in linked question
    – gnat
    Commented Oct 13, 2014 at 7:56
  • @gnat - I think you should double upvote it or downvote it (vote and then retract). A single upvote or downvote penalizes the poster for a bad audit. If its not appropriate, the the poster should not be penalized. When double voting it the first upvote (or downvote) removes it from the audit pool; and the second undoes the unwarranted penalty.
    – user173448
    Commented Oct 25, 2014 at 2:27
  • @jww frankly, if I find that penalizing poster isn't appropriate, I simply let it go and don't vote. In cases like this, audit failure seems more or less fair
    – gnat
    Commented Oct 25, 2014 at 11:54
  • You can't do anything positive or negative to a "known bad" answer in some situations. Perhaps if the answer was deleted or closed? All I know is that I couldn't interact with the "Late Answer" audit I failed.
    – user245382
    Commented Jan 22, 2015 at 23:10
  • @Houseman this is only an issue if post deletion is not guaranteed to be solid (eg deleted by 6 random 2K users from LQ queue) - and there is a request to fix this, by allowing only mod deletions to feed into known bad audits. If one wants to dispute audit failure on mod-deleted post, they better challenge it at per site meta, because that would be not so much challenging an audit but disputing moderator decision...
    – gnat
    Commented Jan 23, 2015 at 10:46
  • ...FWIW I discussed differences between known-good and known-bad audits in more details here: "selection of known bad audits actually involves more than... votes... in order to be deleted, post has to go through quite stringent process..."
    – gnat
    Commented Jan 23, 2015 at 10:52
  • 1
    I've just fallen fowl of what I believe to be a "bad audit", and the question was deleted, so I have no further action I can take.
    – Relequestual
    Commented Jul 6, 2015 at 10:53
  • This answer does not even address the question. The question is about the problem of poor reviews entering the automated audit system. Your answer is only about how to fix specific instances after they already caused damage.
    – jogojapan
    Commented Jan 1, 2017 at 10:07
  • wonder if you read first two comments under this answer, those voted up 11 and 4 times @jogojapan
    – gnat
    Commented Jan 1, 2017 at 20:49
  • This is a good idea for known good audits that are actually bad, but most known bad audits are deleted, so I can't upvote them.
    – Donald Duck
    Commented Jun 25, 2017 at 22:01
  • 1
    yes @DonaldDuck disputing incorrect known-bad audits is more difficult when these are deleted (may need meta post or flag if you have 10K rep) but if you think of it this difficulty is somehow balanced by less frequent mistakes of that kind. This is because deletion typically involves human verification (by moderator or reviewers or delete voters) - this is of course still error prone to some extent but not as much as in known-good audits which are picked by brainless algorithm that mostly just follows a handful of random upvotes
    – gnat
    Commented Jun 25, 2017 at 22:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .