What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

Background

This is somewhat specific to StackOverflow. My concern is the "I understand" button that appears when you fail a Review Audit.

I had this experience twice recently (after never having previously failed an audit), once shortly before the revamping of the Close/On Hold mechanism and once today.

In both cases (one a Close audit, one a Reopen audit) I'd spent several minutes studying the Question under Review, and would happily explain my reasons for voting as I did. I don't expect the automated audit process to be "perfect". However my post here is to point out the "I understand" button needs an alternative, which might be useful in trapping those Review Audit cases which have significant flaws for this purpose.

Since the StackExchange Community determines standards and moderates itself, "I understand" that the primary purpose of the audit reviews is to discourage superficial "click-through" behavior on the part of members eager to gain the corresponding badges. While it might also serve to educate users like myself who are motivated to make a dent in the backlog of Close votes, the adjuration to "STOP! Look and Listen" doesn't really accomplish this.

Proposal: Add an "I don't understand" button

So, my proposal is to implement an "I don't understand" button. It would do two things, one mildly punitive to ensure sincerity. It would suspend Review Privileges automatically for a week, or until a Moderator sees fit to remove the suspension. It would also give the User who clicks on it a chance to explain the reasons for his/her review choices, instead of placing them in the double bind of having to click "I understand" or leave the field.

Examples

I was reluctant to give specifics of my Review Audit Fails as it might a cry for a personal remedy. Now I think some specifics might help to convey the fact-intensive nature of these failures. My proposal aims to have quick impartial reviews while the Reviewers memory is still fresh (and can best accomodate a lesson-to-be-learned, if appropriate).

The first example was in a Close Vote Review (IIRC qua Not a Real Question) of a Question about current development options for Windows desktop applications. The OP disliked MS "Windows Presentation Foundation" framework but spoke favorably of some older frameworks.

A few Comments under the Question asked for clarification and complained about an erratic and distracting typography (boldfacing numerous tool names/buzzwords), and the OP had not (yet) responded to any of these. I thought that all this happened recently (within the past day).

After studying the Question I concluded that it was worthwhile and could be answered (e.g. with objective information about HTML5 technology). The annoying boldface emphasis and slightly whiny tone, but (1) I didn't assess those as deal breakers and (2) given a bit more time perhaps the OP might undertake to make those changes him/herself. With these considerations in mind, I voted to Leave Open.

Afterward the system disclosed that the Question was actually much older (a month+) and had been Deleted soon after being Closed. Given an opportunity to respond and revise, the OP may have elected instead to Delete their Question.


My second example was a Reopen Vote Review, recently enough to have the new Close Reason framework (apparently as an Off-Topic closure, IIRC). The Question asked about why a Python inequality comparison succeeds in cases involving nonnumeric values. Comments under the Question confirmed the behavior but noted its elimination in Python 3, as well as remarking that why the "feature" existed previously would be open to discussion. [Presumably the Question is still out there on StackOverflow, but searches using both StackExchange and Google did not find the exact item, only duplicates.]

I voted not to Reopen based mostly on two considerations. First, the Question had never been edited, and so not eligible for reopening based on revision. Second, it appeared that the Question was something of a call for mildly opinion-based discussion ("why" does Python 2 behave that way) and closed accordingly.


While these Audit Review cases may well have been constructed programmatically or manually by someone who thought them quite cut-and-dried examples (and I accept the utility of doing something like this), there should IMHO be a feedback mechanism that weeds out those not susceptible to a "teachable moment".

Added: I have discovered a new way to fail a Review Audit, by making (or trying to make) a Comment on (purportedly) a User's First Post. According to the information presented, the 2 hour old Answer being reviewed was being posted more than a week after an Accepted Answer, and it was frankly a little terse in phrasing, so I intended to make a Comment to the effect that when posting Answers so much after the fact, brevity in supplying detail is not so virtuous as it might be when answering in a more timely fashion. The reveal: I'm actually reviewing the tersely phrased but Accepted Answer posted nine days before.

share|improve this question
18  
An idea: The 'I don't understand' button could create a post on meta, or a chat room about the audit where people could discuss it, not simply for moderators to review. Just a thought. –  Richard J. Ross III Jul 14 '13 at 14:30
3  
@RichardJ.RossIII We could even make it a 10k list. –  Undo the Snowman Jul 14 '13 at 14:39
10  
If coders really want to code, I also suggest a "I disagree" button that lets a mod review the audit. Mod agrees with your complaint => as if audit never happened and it won't ever happen again. Mod disagrees => you move towards the ban faster than if you never complained. –  Jan Dvorak Jul 14 '13 at 14:53
26  
@Richard No to Meta - then every person who fails an audit because they legitimately suck at reviewing will throw it over on Meta and we'd end up with a mountain of "why did I fail?" questions. Meta should be reserved as a manual process for good reviewers to ask about audits they legitimately don't understand. Only people who actually need help with an audit will come here, and that's a good thing. –  animuson Jul 14 '13 at 14:54
4  
I disagree. I would understand "I don't understand" as "I don't agree with the audit. Some audits are pick up by random, therefore they can (and sometimes are) be quite controvertial. –  РСТȢѸФХѾЦЧШЩЪЫЬѢѤЮѦѪѨѬѠѺѮѰѲѴ Jul 14 '13 at 17:03
    
@animuson Well, you can always hand them a wall of text with a meta link embedded in it somewhere. –  Manishearth Jul 14 '13 at 19:04
6  
+1, I'd be totally behind this! I think the title of your question could be a bit more direct, though: maybe something like "Add an "I disagree" button for failed review audits"? –  Ilmari Karonen Sep 2 '13 at 14:31
1  
I've been using this as a duplicate so I just wanted to edit to bring out some of the key points, hope you don't mind –  Richard Tingle Oct 15 '13 at 9:57
    
@RichardTingle: I don't mind, your edits break up my "wall of text" in a useful way. –  hardmath Oct 15 '13 at 10:11
6  
I'd vote for a "No, your review failed" button. I've failed several audits recently where the question was clearly off topic, or didn't ask a question at all. It is one thing if I don't understand why the question is good, but completely different when I am still confident the question is bad. –  CoverosGene Dec 13 '13 at 14:55
2  
Hope this happens soon... I just 'failed' a question that was (in my opinion) off topic (suitable for programmers.stackexchange), but the audit thought it was 'high quality'. –  Gus Dec 24 '13 at 18:35
1  
I just hit a question that needed to be closed as a duplicate (a previous commenter had left a dupe link) and got the "look and listen" BS. If this is going to happen frequently, the review queues can kiss my behind. –  Pëkka Dec 26 '13 at 14:55
3  
I just failed an audit because I was about to add a comment on a "good question". However, before I was able to add the comment or really do anything else, I failed. Others had commented on the question. The idea that doing anything other than upvoting a "good" question goes against the idea that this site is based on voting since you can give a "wrong" vote. –  zero298 Apr 5 at 22:24
2  
On the two occassions where I got a bad audit question, I never clicked on the "I understand" button. I just navigated away from the page. It's very condescending anyway, even when it's right. I'd prefer something like "Whatever you say..." –  ouflak Jul 3 at 17:05

3 Answers 3

up vote 229 down vote accepted
+100

Rather than an "I don't understand" button, I could get behind a "This audit is incorrect" one that required you to type an explanation for why you disagreed (with a minimum character limit).

Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review, and would greatly reduce traffic on Meta complaining about audit failures. A disputed audit would not count against a user unless it was reviewed and found to be legitimate.

We'd be able to quite clearly see if someone was abusing this option, and since we have the ability to apply manual review bans, such abuse would not last long.

We all recognize that, outside of the suggested edit audits, there are plenty of bad or borderline audit cases out there. If we can identify and remove those, I think it would improve the overall audit process.

share|improve this answer
17  
Will there be audit review audits for moderators? :P –  Manishearth Jul 14 '13 at 19:04
    
@Manishearth I'd think you could unban yourselves :p –  Undo the Snowman Jul 14 '13 at 20:05
    
@Manishearth don't worry, I am buying rep on the black market, so I can audit you al bwa hahahahahaha (I know, I know, not my finest hour) –  user223277 Jul 16 '13 at 12:22
13  
Note by the way, that the very ability to dispute review audit can be rate-limited in addition to already established rate limits on review. I would consider something like starting with allowing one audit challenge per week (or maybe even per month) for a novice reviewer. For steward reviewer, one challenge a day feels reasonably safe. –  gnat Jul 16 '13 at 13:55
2  
A good idea would be to place the action that failed the audit (downvote for good audit, commenting on good audit, upvoting bad audit, etc.) so that they can place it in context. –  Qantas 94 Heavy Oct 15 '13 at 12:09
1  
It took a while for me to come around to the "This audit is incorrect" version because of the more confrontational tone, but I'd always envisioned typing an explanation for disagreement. I can see the virtue in directly disputing the audit case. –  hardmath Nov 5 '13 at 0:33
11  
Any progress on implementing this? I just hit a borderline one myself (poor but salvageable question, not all that old, named user account) and would have liked the opportunity to explain my reasoning. –  Zack Dec 10 '13 at 4:07
1  
On Mathematics I have just had my first fail after several passes. I think it was a judgment call, yet I was told I was wrong without any chance to engage. I think that fail rate should be taken into account before suspending anyone - everyone will disagree occasionally, and sometimes a consensus needs to be tested. And I also think that the tests need to feed a live conversation about what a good review is. Some of the Maths homework questions get closed before the original poster has a chance to respond to very helpful comments. –  Mark Bennet Apr 22 at 19:59
    
I like this proposal with the exception that a disputed audit should continue to count against the user until resolved. Otherwise, users have an incentive to dispute even legitimate audits. –  Warren Dew May 23 at 6:05
    
@Manishearth, "Will there be audit review audits for moderators?" If there are going to be, I volunteer. –  ouflak Jul 3 at 17:03
    
@MarkBennet: Well, when he gets around to it, his edit will push his question into the reopen-queue, and if he actually resolved the problems with his post, it will be reopened fast. It's working as intended. –  Deduplicator Sep 10 at 18:57

I'm done reviewing anything until there's some sort of automatic disputation option for audits. I don't want to create a meta question for every bad audit either. A number of times, I have opened the offending question or answer in a new window, taken the action I believe is appropriate on it, and moved on. But that's getting old for me.

I have not failed a lot, but the failed audits are very invalidating. Fundamentally I can not click "I understand" on a completely incorrect warning.

As Picard said, "There are four lights!"

I propose a different mechanism that the question or other answer upon clicking "I disagree".

  • go ahead and count the audit as failed (shouldn't happen that often for a careful reviewer anyhow)
  • let me click "I disagree"
  • make me prove I'm not a robot (CAPTCHA'd)
  • let me perform all the actions that I would have performed on the post if it was not an audit or I encountered the post outside the review queue (in my understanding, this would often drop the post out of the audit pool automatically as well)
  • let me click "done" and continue onto the rest of the queue

I think this has the advantages of not being any more gameable by robo-reviewers while still allowing me to do the cleanup that the review queues invite and not psychologically torturing me by making me click "yes" when I believe "no".

share|improve this answer
7  
You make a number of points consonant with my view, not least of which is not creating a meta question for every bad audit. The only further point I'd like to emphasize is that I'm personally open to some kind of learning component/explanatory mechanism when an audit fail seems wrong. E.g. the community standards at StackOverflow need not be the same as at Math.SE, and perhaps some tweaking (of my "collegial" judgements in reviews for the latter) is necessary to do reviews properly at SO. –  hardmath Aug 1 '13 at 16:12
4  
I find the whole concept of review audits patronising and insulting. I used to sometimes go to the review queues, in the interest of helping the Stack Overflow community. I mean, we all play our part, right? But after two "review the reviewer" questions, I decided I don't need the grief. If the community doesn't trust me to review correctly, that's fine. And I will never again review anything. –  David Wallace May 4 at 8:38

Better fix: remove the audits. We may be willing to be digital sharecroppers, but at least do us the courtesy of not 1) wasting our time with fake reviews; 2) insulting us when we "fail" particularly questionable fake reviews.

share|improve this answer
5  
Review audits serve a valuable purpose of reining in robo-reviewers, ensuring everyone is paying sufficient attention. Do you have an alternative solution for that? –  mhlester Apr 22 at 19:59
1  
What is a "robo-reviewer"? I have yet to see a bot account accrue 500 reputation, never mind enough to cast open or close votes... –  user152743 Apr 22 at 20:07
1  
robo-reviewers aren't actually bots, just users approving everything they see in order to get the Steward/Reviewer badges –  mhlester Apr 22 at 20:22
    
@mhlester unfortunately, we still have those even with audits on. –  Jan Dvorak Apr 22 at 20:42
1  
Then I would suggest there are two possible situations: 1) most reviews require careful thought, and "robo-reviewers" are clear outliers that can be dealt with by the mods in the same way as robo-upvoters, downvoters, flaggers, etc. 2) "robo-reviewers" cannot be detected heuristically because most reviews do not require any thought, as "accept" is correct 99% of the time, and the review queue should just be removed entirely. –  user152743 Apr 22 at 20:42
    
My experience is that open/close queues are in the former category, and suggested edits in the latter, though this is on math.se and I have no experience what things are like on the other networks –  user152743 Apr 22 at 20:43
    
@user152743 as for 1) - what would be the way? You can't just reverse everything. Also, you need a way to detect them. 2) is definitely not the case. –  Jan Dvorak Apr 22 at 20:44
    
If 2) is not the case, then it is easy heuristically to identify users who are abusing the system. Flag those users for moderator review, and ban abusers from the review system (or the network entirely). –  user152743 Apr 22 at 20:49
5  
I'd say get rid of the badges. A shame, but the current cure is worse than the disease. I would review out of the goodness of my heart, but I don't anymore because I too think audits are insulting. –  Ben Millwood Apr 27 at 14:36
    
+100 if I could. A shame I posted my comment under the other answer before I saw this one. –  David Wallace May 4 at 8:40
    
I will never understand why some people take it so seriously when they are "insulted" by a computer :D. Same with downvotes and close-votes, virtual stuff. Simply stop taking it personally. –  kapa May 23 at 9:58
2  
@kapa This is not a game. We expect to deal with people here, not AIs. –  Noctis Skytower Jun 9 at 14:29
2  
I also "tried to do the right thing" today and got banned. If no one is worrying that people trying to do the right thing are getting punished, something is wrong. If such people refuse to perform reviews in the future, the system contributes to lower performance. –  Noctis Skytower Jun 9 at 14:55
    
@NoctisSkytower We were supposed to deal with intelligent people. This is not a social site, it is not about egos. It is about programming. –  kapa Jun 9 at 15:28
3  
If you're contributing time and patience on reviewing and you're getting banned for reason you strongly disagree with, then yes - it is offending. There's little intelligence involved in judging. It's a matter of stand. There's no straight line isolating bad posts from the good ones. Just give us an option to circumstantiate our verdicts. –  Tarec Jul 4 at 17:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .