Add an "I don't understand" option for review audits

Question

Background

This is somewhat specific to Stack Overflow. My concern is the "I understand" button that appears when you fail a Review Audit.

I had this experience twice recently (after never having previously failed an audit), once shortly before the revamping of the Close/On Hold mechanism and once today.

In both cases (one a Close audit, one a Reopen audit) I'd spent several minutes studying the Question under Review, and would happily explain my reasons for voting as I did. I don't expect the automated audit process to be "perfect". However my post here is to point out the "I understand" button needs an alternative, which might be useful in trapping those Review Audit cases which have significant flaws for this purpose.

Since the Stack Exchange Community determines standards and moderates itself, "I understand" that the primary purpose of the audit reviews is to discourage superficial "click-through" behavior on the part of members eager to gain the corresponding badges. While it might also serve to educate users like myself who are motivated to make a dent in the backlog of Close votes, the adjuration to "STOP! Look and Listen" doesn't really accomplish this.

Proposal: Add an "I don't understand" button

So, my proposal is to implement an "I don't understand" button. It would do two things, one mildly punitive to ensure sincerity. It would suspend Review Privileges automatically for a week, or until a Moderator sees fit to remove the suspension. It would also give the User who clicks on it a chance to explain the reasons for his/her review choices, instead of placing them in the double bind of having to click "I understand" or leave the field.

Examples

I was reluctant to give specifics of my Review Audit Fails as it might appear as a cry for a personal remedy. Now I think some specifics might help to convey the fact-intensive nature of these failures. My proposal aims to have quick impartial reviews while the Reviewers memory is still fresh (and can best accomodate a lesson-to-be-learned, if appropriate).

The first example was in a Close Vote Review (IIRC qua Not a Real Question) of a Question about current development options for Windows desktop applications. The OP disliked MS "Windows Presentation Foundation" framework but spoke favorably of some older frameworks.

A few Comments under the Question asked for clarification and complained about an erratic and distracting typography (boldfacing numerous tool names/buzzwords), and the OP had not (yet) responded to any of these. I thought that all this happened recently (within the past day).

After studying the Question I concluded that it was worthwhile and could be answered (e.g. with objective information about HTML5 technology). The annoying boldface emphasis and slightly whiny tone, but (1) I didn't assess those as deal breakers and (2) given a bit more time perhaps the OP might undertake to make those changes him/herself. With these considerations in mind, I voted to Leave Open.

Afterward the system disclosed that the Question was actually much older (a month+) and had been Deleted soon after being Closed. Given an opportunity to respond and revise, the OP may have elected instead to Delete their Question.

---

My second example was a Reopen Vote Review, recently enough to have the new Close Reason framework (apparently as an Off-Topic closure, IIRC). The Question asked about why a Python inequality comparison succeeds in cases involving nonnumeric values. Comments under the Question confirmed the behavior but noted its elimination in Python 3, as well as remarking that why the "feature" existed previously would be open to discussion. [Presumably the Question is still out there on Stack Overflow, but searches using both Stack Exchange and Google did not find the exact item, only duplicates.]

I voted not to Reopen based mostly on two considerations. First, the Question had never been edited, and so not eligible for reopening based on revision. Second, it appeared that the Question was something of a call for mildly opinion-based discussion ("why" does Python 2 behave that way) and closed accordingly.

---

While these Audit Review cases may well have been constructed programmatically or manually by someone who thought them quite cut-and-dried examples (and I accept the utility of doing something like this), there should IMHO be a feedback mechanism that weeds out those not susceptible to a "teachable moment".

Added: I have discovered a new way to fail a Review Audit, by making (or trying to make) a Comment on (purportedly) a User's First Post. According to the information presented, the 2 hour old Answer being reviewed was being posted more than a week after an Accepted Answer, and it was frankly a little terse in phrasing, so I intended to make a Comment to the effect that when posting Answers so much after the fact, brevity in supplying detail is not so virtuous as it might be when answering in a more timely fashion. The reveal: I'm actually reviewing the tersely phrased but Accepted Answer posted nine days before.

Added(2): In going through a Close Review Queue, I came to a Question that was clearly being presented as an Audit ("primarily opinion based" being the presumptive reason to consider closing). However the Question was an exact duplicate of a previous Question with an Accepted (and upvoted) Answer, so in my bullheaded way I chose Close as Duplicate (and failed the Audit).

Answer 1

Rather than an "I don't understand" button, I could get behind a "This audit is incorrect" one that required you to type an explanation for why you disagreed (with a minimum character limit).

Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review, and would greatly reduce traffic on Meta complaining about audit failures. A disputed audit would not count against a user unless it was reviewed and found to be legitimate.

We'd be able to quite clearly see if someone was abusing this option, and since we have the ability to apply manual review bans, such abuse would not last long.

We all recognize that, outside of the suggested edit audits, there are plenty of bad or borderline audit cases out there. If we can identify and remove those, I think it would improve the overall audit process.

Answer 2

I'm done reviewing anything until there's some sort of automatic disputation option for audits. I don't want to create a meta question for every bad audit either. A number of times, I have opened the offending question or answer in a new window, taken the action I believe is appropriate on it, and moved on. But that's getting old for me.

I have not failed a lot, but the failed audits are very invalidating. Fundamentally I can not click "I understand" on a completely incorrect warning.

As Picard said, "There are four lights!"

I propose a different mechanism that the question or other answer upon clicking "I disagree".

• go ahead and count the audit as failed (shouldn't happen that often for a careful reviewer anyhow)
• let me click "I disagree"
• make me prove I'm not a robot (CAPTCHA'd)
• let me perform all the actions that I would have performed on the post if it was not an audit or I encountered the post outside the review queue (in my understanding, this would often drop the post out of the audit pool automatically as well)
• let me click "done" and continue onto the rest of the queue

I think this has the advantages of not being any more gameable by robo-reviewers while still allowing me to do the cleanup that the review queues invite and not psychologically torturing me by making me click "yes" when I believe "no".

Answer 3

Qualities of a good Audit

• 90%+ people pass without problem
• 50%+ of people who fail understand why

Qualities of a bad Audit

• 50%- people fail (might as well let a nickle do my reviews!)
• 50%+ of people who fail don't understand why (The Audit is either wrong, or has an ambiguous answer)

In general, I think the audit system should be able to moderate itself. By tracking an audit's history, the system should implicitly know the quality of an audit. If too many people are failing an audit, it should be removed auto-magically (no mods or review ques needed). At the same time, there REALLY should be a "I don't understand" option, and/or the option to dispute the audit.

As is, people failing audits are posting Meta's about why/remove the audit because they think (and it is) the only thing they can do to improve the audit system.

Other things that could be done to help moderate the audits is

• An option to vote on audit quality
• An option to dispute bad/low-quality audits
• Dropping audits that nickles are better at passing (and after x uses).
• An audit queue (where new audits go to be approved before entering the system)

But at the very least, SOMETHING has to be done.

Answer 4

Better fix: remove the audits. We may be willing to be digital sharecroppers, but at least do us the courtesy of not 1) wasting our time with fake reviews; 2) insulting us when we "fail" particularly questionable fake reviews.

Answer 5

I'd like to add here my vision (transferred from meta SO), maybe this 6-year old question is re-visited by anyone having the power to trigger the change.

For many reasons, it would be handy to have an option/feature which could allow reviewer to dispute review audits whenever they find them inappropriate right on the spot, by clicking link or button, without having to resort to making a new topic on meta. Introducing such feature could boost the process of polishing review audit system to help filter it from bad audits.

Advantages (summed up)

Much more users involved

1. Making one click is faster than making up a new topic on Meta.

2. Making one click is an opportunity for users not registered on Meta to report disputed audit they may not even aware it has been possible.

As a positive side effect, this can potentially reduce amount of disputed audit topics on meta. Those who never took part in audits improvement, get involved; and those who did it via meta topics do it silently.

Voting beyond personal interest

It is common for people to come here to dispute audits they were involved with, e.g. to lift a review ban. Being able to vote for an audit goes beyond personal interest. You don't necessarily vote because you were involved, you vote to improve the system, and it costs next to nothing.

Possible solutions, restrictions

One of the possible restrictions is to make this link (button) visible only for users, possessing the Reviewer badge (for any queue). This will temporarily restrict dispute reports from review starters.

There are few options on how to carry this system. One of the options is to make it collect votes, and once there are enough, the particular audit is removed.

The link (or button) should be visible regardless of the result of the review audit (passed or failed), right after it is complete:

Placing small link beneath the main text, stating something like "report disputed audit" (discussable) would be Ok in my opinion.

Another variant is to introduce two-way voting (up/down) options for audits to rate them. This way we have medium. And once the rating is enough, review audit is also removed.

Finally, as Tezra wrote, at the very least, SOMETHING has to be done.