What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 130 Stack Exchange communities.

Some people are complaining about the automatically generated audits. They are based on close reasons and apparently are a little controversial :-)

I propose that for "good reviewers" (definition TBD) an extra UI component be added to let the reviewer suggest this as an audit item. Possibly:

  • Reject and add as audit button (Close and add as audit, etc)
  • checkbox next to Reject, Close, etc
  • checkbox on the Reject reason, Close reason etc dialog (could get messy, and some actions such as Reopen don't involve a subsequent dialog)
  • dialog after rejecting, closing, Recommend Deletion, etc

Further, after passing or failing an audit, we could ask "good reviewers" "was this a good audit"? Bad audits, whether auto-generated or human suggested, could be removed from use if enough "good reviewers" identified them as such.

Now, what's a good reviewer? I don't want to judge only on behaviour in the review queue. I think flagging well is a good sign. So is editing - maybe an edit-related badge? And yes, some number of reviews would be good. Maybe some ratio of pass to fail on review audits. But not JUST what we do in the queues.

(I first suggested this in my answer to http://meta.stackexchange.com/a/159573/147247 but I'm bringing it up as a feature request of its own and to apply to all the queues.)

share|improve this question
2  
Hmm, if we can't have decent audits to trap bad reviewers, then how are we going to come up with a system to detect good ones? ;) –  Bart Feb 20 '13 at 19:25
4  
@Bart - that's why behaviour outside the queues is key. Someone with 1000 helpful flags (or probably with any), with Copy Editor, with Yearling, etc is probably someone who is a "good reviewer" if you add some review accomplishments to the list. Someone with Steward in one queue and nothing else - probably grinding the remaining queues for more Stewards –  Kate Gregory Feb 20 '13 at 19:28
7  
Agreed. Or why not take this further and have good reviewers review reviews, instead of audits? –  Pëkka Feb 20 '13 at 19:43
27  
Then we'll need review reviewer reviewers reviewing review reviewers, @Pekka. –  Shog9 Feb 20 '13 at 20:08
8  
Quis custodiet ipsos custodes? –  David Robinson Feb 20 '13 at 20:39
11  
@Shog why? That's a tired argument and I don't buy it. What are the actual chances of 3 site members in good standing wrongly deciding that somebody else's bad review is good, or good review is bad? And even if that happens once, what are the chances of it happening a second time to the same person? Reviewing reviews would also lead to every, or at least many review actions being judged, as opposed to occasional DUI checks as the audit system does. –  Pëkka Feb 20 '13 at 21:35
2  
@Pekka웃 agree - especially if we give no incentive (especially not "ok then you didn't fail after all") for reviewing an audit, and we offer the capability only to "good reviewers" –  Kate Gregory Feb 20 '13 at 21:51
5  
Really, @0A0D? Nothing about this looks a bit... destructive to you? –  Shog9 Feb 21 '13 at 23:07
2  
You know how tricky the datasources car part can be @Shog9... –  Kate Gregory Feb 21 '13 at 23:14
1  
@Pekka: in all seriousness, I think there's a place for that - but routinely subjecting all reviews to meta-reviews would be a ton of overhead. A more practical model is one that uses some combination of user input (flags, editing, voting) and heuristics to identify areas that need further review. –  Shog9 Feb 22 '13 at 3:38
3  
I would love to get this one added to the list :) –  Jack Feb 22 '13 at 10:32
4  
I don't agree with the closure of this question as a duplicate of Review audits and “I understand” button. True, both questions propose a similar mechanism to provide feedback about whether an existing audit is good/bad. However this question also proposes a mechanism (which involves regular users) to create new audits... –  Old Checkmark Jul 22 '13 at 12:02
1  
... While the proposed mechanism to create new audits is similar to the proposed mechanism to review existing audits, the goal is opposite and as such has different advantages and drawbacks. I think this would mean that the 2 mechanisms could be considered somewhat separately. An analogy would be: "I want to use tool A to do B and C" (this question) , the other question is about "I want to use tool A to do C". –  Old Checkmark Jul 22 '13 at 12:07
6  
Did you know you get no notification if your question is closed? Thanks @doubleDown for commenting. I don't see "let humans create new audit entries" as a dupe of "let humans say they didn't like the code-chosen audit entry they were shown" at all. –  Kate Gregory Jul 22 '13 at 12:46
2  
The problem is, that the criteria for off-topic questions have changed and many 'quasi-good' questions from past are terrible questions now. Many community members are still thinking with old criteria. Unter those circumstances blindly chosing autits basing on upvotes is simply not a good idea. –  РСТȢѸФХѾЦЧШЩЪЫЬѢѤЮѦѪѨѬѠѺѮѰѲѴ Nov 29 '13 at 7:45

1 Answer 1

While there is no "officially implemented" solution for this, one can use whatever means are at their disposal now in order to bring the "human factor" to audits selection.

When you spot a slippery audit, go straight to the "item" it uses and do the action opposite to audit direction.

  • If you feel something rotten before submitting the audit, use link to the audit item to get there for corrective action. If you found that you were screwed after the audit, bad audit and item can be found in the activity tab in reviews subsection.

If the audit item has been wrongly served as "known good", down / close vote it. If it was pretending to be "known bad" against your judgement, vote it up / reopen.

I always do this to audits I disagree with.


As I typically open the items in queue in separate tab (for more thorough review), it often happens that I spot slippery audit and perform "correction" even before completing review. It feels somewhat weird to click Looks OK at the item you just downvoted but oh well. I am not going to decrease my "audit weight" just because of a mistake in automatic selection algorithm, and knowing that reviewers after me won't get into this trouble anymore makes it less painful.

share|improve this answer
2  
This does nothing to help when you are banned from the review queue for 'failing' audits despite having less than 1% disputed or rejected reviews. –  FerretallicA Oct 13 at 7:41
1  
@FerretallicA agree, my approach is only a workaround that merely saves next reviewer from getting unfair suspension. Real solution would be to either introduce human verification of reviews as suggested here or allow challenging and invalidation of wrong ones as suggested in linked question –  gnat Oct 13 at 7:56
    
@gnat - I think you should double upvote it or downvote it (vote and then retract). A single upvote or downvote penalizes the poster for a bad audit. If its not appropriate, the the poster should not be penalized. When double voting it the first upvote (or downvote) removes it from the audit pool; and the second undoes the unwarranted penalty. –  jww yesterday
    
@jww frankly, if I find that penalizing poster isn't appropriate, I simply let it go and don't vote. In cases like this, audit failure seems more or less fair –  gnat 16 hours ago

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .