What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 130 Stack Exchange communities.

It is currently too easy for robo-reviewers to bypass the honeypots for the review audits.

Steps:

  1. Go in the Review section and in the First posts section (or any of them where review audits are enabled)
  2. If you click on the Add comment link and that is an audit, then you will pass the audit and get the "Congratulations" message.
  3. If it's not an audit, you can just click on "Nothing to do".

This seems to me a very easy to find loophole which could be used by robo-reviewers, and shouldn't even delay them for more than half a second. If they do it for every single review then they are guaranteed to pass all audits.

Note that this also works if you click on "Flag" for example, and I suspect pretty much any link appearing on the review.

I would propose that you get the "Congratulations" message only after you get to a final irreversible action, for example clicking on "Flag" AND defining the type of flag, or clicking on "Close" AND choosing the type of close, or any other method that would be considered irreversible. The cost of doing such an irreversible action should be null in case of audits, for example if you flag a post, this shouldn't count towards the daily flag cap.

Tested on this review.

share|improve this question

1 Answer 1

up vote 12 down vote accepted

First off, this bothers me too. Unfortunately, there are actually a few different ways to "cheat" these, and fixing them would require leading on honest reviewers rather a lot - consider someone spending serious time to try cleaning up a train-wreck of a post only to be told that their edit should be discarded. Even for comments, wasting someone's time writing is probably not going to be received well.

In the end, these audits are intended to catch folks who really aren't paying any attention at all.

Besides... There's actually a much bigger give-away for a "robo reviewer" than failing audits. If you're taking the exact same action on every post you come across, regardless of length, topic, or others' reaction to it... You're probably not actually reviewing anything.

Either way, such folk will tend to find themselves blocked from /review for a while.

share|improve this answer
3  
+1, you could implement some kind of classifier to train on an history of people who are identified as robots or people who are identified as real reviewers, and then classify every reviewer based on his history. I'm curious what the rate of false positives would be though and if that would be a viable solution. –  Charles Menguy Jan 26 '13 at 2:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .