What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 130 Stack Exchange communities.

I have noticed this "STOP! Look and Listen" message when the system audits you by having you review something to check how you respond.

However, this audit came up and I was tricked. I wanted to up vote the existing comment as well as provide my own. But there was no up arrow for the comment. This completely threw me off and had me thinking something was wrong. To check if any of the interface was working, I went to up vote the question. BAM I am hit with the "You didn't pass" message.

Therefore the audit itself is the cause of this. This is broken.

EDIT: After running into my 3rd audit (which I passed), there are two pieces that are broken. First, I should be able to up vote comments or else I can detect it is an audit. Second, I am told if I pass/fail BEFORE clicking "I'm Done" giving me no chance to undo an upvote or to perform 2 actions.

EDIT 2: Another problem that is caused by the system reacting before the user clicks "I'm Done" is they can click down vote for every single review. If it is an audit, it ends and they passed. If it is not an audit, they can change their vote to an up vote. While this sounds like a lot of work, it really only takes a second longer and ensures you will never fail an audit.

share|improve this question
10  
Why would you choose to click the upvote button rather than the downvote button? –  animuson Nov 28 '12 at 6:03
10  
@animuson I was clicking anything to check if the interface was working. I thought my browser was misbehaving. –  Austin Henley Nov 28 '12 at 6:04
30  
If you were piloting an airplane full of passengers and your controls locked up. Would you press the self-destruct button to see if it worked? :-P –  Mysticial Nov 28 '12 at 6:07
67  
@Mysticial Not relevant. The up vote button can be EASILY undone. Up/down made no difference to me while only checking if the interface was working. –  Austin Henley Nov 28 '12 at 6:09
11  
You obviously missed the sarcasm... –  Mysticial Nov 28 '12 at 6:11
9  
You took precisely the action the interface was programmed to dislike .. I'm not sure how that could be done any differently? As far as I know you can't up vote comments through review (at all), I don't believe it's a giveaway. Checking on that though, ICBW. –  Tim Post Nov 28 '12 at 6:19
15  
@TimPost Wrong, you CAN up vote comments through review. Also, it did not give me the ability to undo the up vote so how is it realistic in any way? It isn't like I clicked "I'm Done" –  Austin Henley Nov 28 '12 at 6:20
3  
@AustinHenley You're right, (I did say ICBW) - I was trying to reproduce in the wrong queue. –  Tim Post Nov 28 '12 at 6:23
    
Does the interface behave like this on non audits? (I have only reviewed Questions/Answers without comments so far) –  Richard Tingle Jul 16 '13 at 13:08
    
My complaint is with late answers where an existing answer is shown with a User55555 type new (bogus?) author. I looked at the page and thought someone had just copied and pasted the existing good up-voted answer. So I go to flag and get a 'Stop! Look Listen' audit failure. –  ficuscr Aug 7 '13 at 0:01
3  
@ficuscr I know exactly what you mean. In fact, I made a post about the missing username just the other day. –  Austin Henley Aug 7 '13 at 2:51

2 Answers 2

up vote 80 down vote accepted
+50

I think what you discovered is a mistake in the design of "honeypot" test.

This mistake is likely based on the wrong assumption1 that test should catch anyone who randomly clicks on stuff deemed incorrect. The assumption looks incorrect because per my understanding, "honeypot" is intended to catch "fake reviewers" - those abusing review actions with the purpose (P-U-R-P-O-S-E) to increase their review count.

  • The only way to increase review count in First Posts and Late Answers queues is to click specific button(s) like I'm Done or No Action Needed2 - as long as this did not happen, it is wrong to assume test completed.

Note that current test design doesn't fit well to another officially stated goal of review audits: "help new reviewers hone their moderation skills" - unless, of course, a fear of doing easily reversible actions is considered a moderation skill worth "honing".


Mistakes like you have seen jeopardize the intent of the test and open the door to claim it invalid.

A reasonable way to address this issue has been recently proposed here:

...I could get behind a "This audit is incorrect" [button] that required you to type an explanation for why you disagreed (with a minimum character limit).

Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review, and would greatly reduce traffic on Meta complaining about audit failures. A disputed audit would not count against a user unless it was reviewed and found to be legitimate.

We'd be able to quite clearly see if someone was abusing this option, and since we have the ability to apply manual review bans, such abuse would not last long...

Note by the way, that the very ability to dispute review audit can be rate-limited in addition to already established rate limits on review.

I would consider something like starting with allowing one audit challenge per week (or maybe even per month) for a novice reviewer. For steward reviewer, one challenge a day feels reasonably safe.

share|improve this answer
13  
Exactly, not being able to click "I'm Done" is the biggest fault with it at the moment and it really does defeat any validity in the audit. –  Austin Henley Nov 28 '12 at 7:09
6  
Waiting until all actions are performed for an audit task wastes valuable time that could be spent reviewing an "actual" review. Audit tasks are only concerned with the first impulse action taken on a post. –  Geoff Dalgas Nov 28 '12 at 7:25
18  
@AustinHenley yup. You'll likely be told "test teaches you to think before you click" but that's a bullshit. The only moment when this would be appropriate is at clicking I'm Done - because it is the moment when "fake reviewers" reveal their abuse. Everything you do prior to that is just irrelevant for honeypot test purpose –  gnat Nov 28 '12 at 7:27
5  
@GeoffDalgas no matter what, completion of the test before I'm Done is pressed supports claims that test missed the target - "fake reviewers". We already have review system that was intended to improve quality of the posts but somehow missed that target –  gnat Nov 28 '12 at 7:30
14  
@GeoffDalgas You must not review much on a mobile device. I click the wrong arrow very, very, very frequently. –  Austin Henley Nov 28 '12 at 7:30
7  
There's nothing being done with these audits at the moment as far as negative consequences. It's only meant as an aspect of teaching people to do the right thing such as not upvoting an obviously wrong post which was the #1 complaint for review thus far. It gives us an opportunity to inform those who make an impulsive or misclicked action on an audit task about the "right" thing. –  Geoff Dalgas Nov 28 '12 at 7:35
6  
@GeoffDalgas see above. If you want to teach, pick an appropriate moment for that. Current design rather fuels confusion than teaches. –  gnat Nov 28 '12 at 7:39
4  
@GeoffDalgas It just doesn't seem very useful to me since it isn't accurate (and for example, it prompted this post instead of informing me anything) Why does it seem like you are content with how it works instead of improving it using the things brought up here? –  Austin Henley Nov 28 '12 at 7:40
8  
@GeoffDalgas if a proper audit can not be performed at the moment of clicking I'm done, consider dropping that audit completely. As I mentioned, this can really do less harm than jeopardizing it the way it is now –  gnat Nov 28 '12 at 7:48
1  
@GeoffDalgas I think that is a cost of an audit system like this, if it continues to exist. To the reviewer, it isn't a big deal because they could put in all that work just for the post to be deleted. It makes no difference from the reviewer's perspective. –  Austin Henley Nov 28 '12 at 7:49
4  
If the reviewer passes an audit, is it necessary to even inform them? Let them think it was real and add 1 to their review count :) –  Austin Henley Nov 28 '12 at 7:51
3  
@GeoffDalgas Informing them when they fail isn't enough? Why should it affect real reviewers. –  Austin Henley Nov 28 '12 at 7:53
4  
@GeoffDalgas I think that is invalid since it falsely tells me I am doing something wrong. –  Austin Henley Nov 28 '12 at 8:02
4  
@AustinHenley audit tasks are currently working in providing the right guidance to those who make an impulsive call on a post based on the data we have collected thus far. If you don't happen to select the correct action initially on an audit task - don't worry! We aren't doing anything with this data. The thing we are trying to combat at the moment is mindlessly upvoting posts in review to do nothing more than increase review stats which are harmful to review system. –  Geoff Dalgas Nov 28 '12 at 8:12
5  
@GeoffDalgas I think you're out of date now. I can't agree with "the problem is that a single person could waste a lot of time performing all of the actions possible" because there just isn't much in the review queues (apart from close vote queue) for it to matter. It is not urgent that reviewers review a real post any more, it is urgent that reviewers review well. Please test the full review, not the (perhaps accidental) first click. –  AndrewC Jan 31 '13 at 9:59

While this sounds like a lot of work, it really only takes a second longer and ensures you will never fail an audit.

We're implementing both automated and manual checks for this. Remember: all review actions are tracked, so fooling the simple automated checks isn't going work for long. And if we find folks going out of their way to abuse this, chances are they'll be blocked from /review for their troubles.

Just try and "do the right thing" and you won't have anything to worry about.

share|improve this answer
10  
I am not worried. I was notifying everyone of the bug and the broken audit mechanism. –  Austin Henley Nov 28 '12 at 7:06
2  
So the honeypots will always be on opposite extremes, either really good or really smelly? The only thing that comes to mind is a post where someone might change their vote after reading again. I'm sure there's a limit to how specific you can be, though. –  Tim Post Nov 28 '12 at 7:06
1  
The intent is to also not waste a reviewers time with reviewing in detail any post that is an audit. If an audit is being performed make it as swift as possible, give guidance, and move on to something more important @AustinHenley. –  Geoff Dalgas Nov 28 '12 at 7:10
7  
@GeoffDalgas False guidance, proven by my scenario in this post. It needs to be displayed exactly as if it wasn't an audit: comments should have up arrows and anything else that is missing. –  Austin Henley Nov 28 '12 at 7:12
3  
Comments are too much of a variable for audit tasks - don't expect they will be there in future audit tasks @AustinHenley. –  Geoff Dalgas Nov 28 '12 at 7:22
5  
I'd rather not get "tripped up" during review when I downvote a highly positive post. Upvotes do not necessarily mean correctness or quality. –  user7116 Nov 28 '12 at 19:45
13  
I think I would be much more upset if an audit task told me I was doing the wrong thing when I knew I wasn't then for me to "waste" time performing a quality review on a fake item. I don't consider it time wasted; I consider it time spent informing the system of my value as a reviewer. If every other review was an audit maybe it'd be a problem, but as long as it's not too too often it's most likely a cost worth paying. Now if the audit stopped me early to say, "clearly you're on the right track, you can stop as this isn't real." that's fine. The problem is stopping early to say you're wrong. –  Servy Nov 28 '12 at 19:54
4  
High-quality posts are a LOT harder to agree upon than low-quality ones. I don't see how you can do something like that without a lot more false positives. –  Justin Morgan Nov 28 '12 at 22:19
    
@TheGuyWhoDisagreesWithYou Well, you could build in an acceptable fail rate. It might be acceptable to downvote 5-10% of posts with a score of 5+ and a checkmark, but if you have a 40% downvote rate on such posts something might be up. As long as a reasonable margin of error has no negative consequences I'll reserve judgement until specifics are given. –  Servy Nov 28 '12 at 22:35
6  
Review is no longer urgent with so many empty queues. Wasting reviewers time is not an issue. Quality of review is now the main issue. –  AndrewC Jan 31 '13 at 10:02
2  
@GeoffDalgas unfortunately "trying to do the right thing" still leads to bans from reviewing even if audits are faulty. I "tried to do the right thing" today and got banned. The system is inherently faulty because it is so black and white. –  CDub Dec 6 '13 at 1:07
    
I also "tried to do the right thing" today and got banned. If no one is worrying that people trying to do the right thing are getting punished, something is wrong. If such people refuse to perform reviews in the future, the system contributes to lower performance. –  Noctis Skytower Jun 9 at 14:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .