"STOP! Look and Listen" audit tricked me

Question

I have noticed this "STOP! Look and Listen" message when the system audits you by having you review something to check how you respond.

However, this audit came up and I was tricked. I wanted to up vote the existing comment as well as provide my own. But there was no up arrow for the comment. This completely threw me off and had me thinking something was wrong. To check if any of the interface was working, I went to up vote the question. BAM I am hit with the "You didn't pass" message.

Therefore the audit itself is the cause of this. This is broken.

After running into my 3rd audit (which I passed), there are two pieces that are broken. First, I should be able to up vote comments or else I can detect it is an audit. Second, I am told if I pass/fail BEFORE clicking "I'm Done" giving me no chance to undo an upvote or to perform 2 actions.

Another problem that is caused by the system reacting before the user clicks "I'm Done" is they can click down vote for every single review. If it is an audit, it ends and they passed. If it is not an audit, they can change their vote to an up vote. While this sounds like a lot of work, it really only takes a second longer and ensures you will never fail an audit.

Answer 1

I think what you discovered is a mistake in the design of "honeypot" test.

This mistake is likely based on the wrong assumption¹ that test should catch anyone who randomly clicks on stuff deemed incorrect. The assumption looks incorrect because per my understanding, "honeypot" is intended to catch "fake reviewers" - those abusing review actions with the purpose (P-U-R-P-O-S-E) to increase their review count.

• The only way to increase review count in First Posts and Late Answers queues is to click specific button(s) like I'm Done or No Action Needed² - as long as this did not happen, it is wrong to assume test completed.

Note that current test design doesn't fit well to another officially stated goal of review audits: "help new reviewers hone their moderation skills" - unless, of course, a fear of doing easily reversible actions is considered a moderation skill worth "honing".

---

Mistakes like you have seen jeopardize the intent of the test and open the door to claim it invalid.

A reasonable way to address this issue has been recently proposed here:

...I could get behind a "This audit is incorrect" [button] that required you to type an explanation for why you disagreed (with a minimum character limit).

Such a button would remove an audit case from rotation pending review by devs (or moderators) in a special list. These cases seem like they'd be quick for us to review, and would greatly reduce traffic on Meta complaining about audit failures. A disputed audit would not count against a user unless it was reviewed and found to be legitimate.

We'd be able to quite clearly see if someone was abusing this option, and since we have the ability to apply manual review bans, such abuse would not last long...

Note by the way, that the very ability to dispute review audit can be rate-limited in addition to already established rate limits on review.

_{I would consider something like starting with allowing one audit challenge per week (or maybe even per month) for a novice reviewer. For steward reviewer, one challenge a day feels reasonably safe.}

Answer 2

While this sounds like a lot of work, it really only takes a second longer and ensures you will never fail an audit.

We're implementing both automated and manual checks for this. Remember: all review actions are tracked, so fooling the simple automated checks isn't going to work for long. And if we find folks going out of their way to abuse this, chances are they'll be blocked from /review for their troubles.

Just try and "do the right thing" and you won't have anything to worry about.