521

Rather than an "I don't understand" button, I could get behind a "This audit is incorrect" one that required you to type an explanation for why you disagreed (with a minimum character limit). Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review,...


266

What is the purpose of this test? The test (known as a review audit) is designed to help new reviewers hone their moderation skills, while nudging more experienced users who don't seem to be paying close attention to what they're reviewing. Some people were in the habit of up-voting everything they reviewed without regard to quality; extremely low-quality ...


108

I'm done reviewing anything until there's some sort of automatic disputation option for audits. I don't want to create a meta question for every bad audit either. A number of times, I have opened the offending question or answer in a new window, taken the action I believe is appropriate on it, and moved on. But that's getting old for me. I have not failed a ...


98

Review queue audits are now in place. A lot of people are blindly upvoting everything in the queue without even reading the posts, just so that they can get the badges. So, some posts in the review queue are designed to catch such users, these are known as "audits". You failed too many, so you're review-banned now. If you're in it for the badges/stats, don'...


96

I think what you discovered is a mistake in the design of "honeypot" test. This mistake is likely based on the wrong assumption1 that test should catch anyone who randomly clicks on stuff deemed incorrect. The assumption looks incorrect because per my understanding, "honeypot" is intended to catch "fake reviewers" - those abusing review actions with the ...


93

You caught me. We throw a few known cases into the queue* as "gotchas" to keep people from just nailing "Looks Good" or "Close" or whatever without reading the questions. If you see one of these just handle it normally, we're probably going to make them a little more... subtle over time. For now we're pulling from really obvious sources: spam, and ...


82

We've rolled out audit tasks for suggested edits in our more recent builds. What we're doing here is actually kind of fun. They're also sometimes hilarious. Since we know suggested edits have really noisy history, the approach we use for other queues (selecting "known good" or "known bad" content to then fake numbers on) won't really work. Instead, we're ...


71

I think it should be possible to add a suggest as honeypot button to all the items in all the queues. Maybe only to those with a certain rep, or certain number of helpful flags. I have come across any number of egregiously bad edits, which I've linked to before. If you wanted to construct some, may I suggest: add a peripherally related tag while leaving ...


66

This has been implemented and deployed in build 2013.7.26.1279 (meta) / 2013.7.26.898 (sites). The triggering mechanism was not changed. It can be turned on or off on each site. When turned on it works like this: each review ban counts (even manual ones made by moderators). a 30 day window is used 1st ban within the window -> duration: 2 days 2nd ban ...


66

I understand the problem with people cheating the first post review audits by simply clicking the add comment button, but there really has to be a more graceful way to handle it. It should be as simple as forcing the person to actually write the comment and submit it which will make it less abusable, and if they are abusing the system, they'll be caught ...


62

While there is no "officially implemented" solution for this, one can use whatever means are at their disposal now in order to bring the "human factor" to audits selection. When you spot a slippery audit, go straight to the "item" it uses and do the action opposite to audit direction. If you feel something rotten before submitting the audit, use link to ...


56

Yes, please. I was about to type up a request for the same. We have a bit of a problem right now on Stack Overflow, as the Winterbash hat for reviews has caused a bunch of new reviewers to all of a sudden jump into the review queue. Many (most?) of them are just in it for the hat, so they approve everything they see. This is leading to a lot of really ...


52

I couldn't agree more. This is impersonation in an explicitly negative scenario, which amounts to libel if you look at it too closely. I thought it was kinda creepy when I first figured it out (which took a while), but got used to it. You've reminded me that I shouldn't have done that. On a practical note, if you're a user who accepts edits blindly (which ...


52

I like the idea of marking bad reviews, because right now I either have to keep a "naughty" file with a list of these, or permanently annotate someone's account with them. There's one specific case where this could even be automated. For answers in the "Late Answers" and "First Posts" queue, I would like some sort of marking or identification when a user ...


46

Actually, we're already doing some of this - if you spend some time in the Low Quality queue, you'll almost certainly encounter a "honeypot" post sooner or later - one that's been overwhelmingly judged by the community as high or low quality. If you respond incorrectly, you get a warning - and the failure is logged. We're not currently using this for ...


46

I fully support this proposal. The main reasons why it makes sense to me are: Spam does not belong in the LQ review queue in the first place. As far as I understand, spam flags are handled by diamond mods, and do not send posts to a review queue. Therefore, spam vs. not spam decisions are not a normal part of working the LQ review queue. Related to the ...


41

My usual flow when reviewing is: Read the title Read the question Click the "Close" button, skim trough the reasons if I am not sure and verify that what I just read is okay with all the points. Either click "X" to close the closing window and Leave Open or Skip, or choose the close reason and close. To clarify; I am not interested in what other ...


40

If the review audit didn't even wait to see if you unchecked the "suggested edit was helpful" box when improving the edit, this seems like a design flaw; of course you should improve it if you see anything in the original post that should be edited, even if the suggested edit is a bad one. Are you expected to wait for 2 more reviews and then remember the ...


38

Ouch, my fault... I accidentally set a flag that made everything an audit. I meant to do this locally (not on prod!) and somehow didn't notice my mistake until this meta thread was brought to my attention. I am updating our logic to make sure this flag is completely ignored on prod, so this doesn't happen again. Sorry about that.


36

Starting the other month, review audit checks were added to the review queues. These were added after disturbing "rubber stamping" was noticed, which resulted in poor quality content being upvoted constantly. These audit checks are questions that are known to be high-quality, or known to be low-quality. It is expected that you, as a reviewer, are able to ...


35

However, it kinda falls apart when you want to close a known-good question as a duplicate. Fixing this would require dragging in code that's currently being extensively re-written; doing so is not feasible at this time. It may or may not become possible in the future - even without the current re-write, the logic for that UI isn't trivial to ...


34

Bill is a moderator and likes to use /review as a tool for moderation. Bill gets an unlimited number of close votes, because he's a moderator. Bill's use of /review for moderation tasks has prompted us to create a special interface for moderators there, which we hope to have available in probably six to eight weeks. It just naturally lends itself to ...


34

I just want to take this opportunity to make sure everyone knows about the close vote review filter tool. It makes blowing through 40 reviews pretty quick and easy, so it's one of my favorite new features of the past year. I'll often filter reviews on problem tags like facebook or android, or on a tag that someone brings up for burnination here on Meta. ...


33

No, they are not. "Skip" does one thing: It removes the page from that review queue for you. That's it. If you "Skip", you mean that "I don't wanna review this". Since "reviewing" includes reading the post, there's no need to punish you for not acting on a post you may not have read. The audits are to stop robo-reviewers (who do it for the badges) from, ...


33

We didn't show positive feedback on these initially, for more or less the reasons you state. It confused people. I don't think removing audits entirely is ever a good idea. That said, we could certainly show them less often for folks with a good track record... and more often for folks with a bad one. What do you think? Maybe, 100% success rate for, say, ...


33

One thing that we've been talking about amongst the moderators is a way to identify reviewers who approve really troublesome content, even after the fact. Anyone who does anything other than flag obvious spam or trolling is harming the site by letting terrible content survive the review process. When a post receives a spam, offensive, or not-an-answer flag ...


31

Woops, that's not good. I suppose we always suspected this would come back to bite us, but perhaps not so spectacularly. Showing fake users on audits is non-ideal, because it creates an easy way to quickly detect audits: Does the display name look fake? Even if not, does clicking the display name bring me to a real user profile? We just pushed a change (...


29

I think this would only result in more pain. We'd basically be telling people: Because you failed x out of n audits, every review you've ever made is bad That could conceivably be the case in some instances, but weigh the hassle of reviewing every single action the user took against the hassle of some posts that were inaccurately reviewed. It seems like ...


29

I just ran a query and want to be clear - until 3:05 UTC today, the question had a positive score of 5 with no down votes, no moderator flags, no close votes and had been marked as a favorite. The question is also well over a week old, has an up voted accepted answer and hasn't moved prior to today. Everything negative associated with that question happened ...


29

I understand the purpose of positive audits but I think the system is flawed. The good way to review is take time, read the post and understand it. When doing that, I find myself having all sorts off thoughts around the topic mentioned in the post. Sometimes I would like to add something. Sometimes I know a good link that would add to the post. Sometimes I ...


Only top voted, non community-wiki answers of a minimum length are eligible