-9

During the last few days, I get at least one computer-generated vandalism review (a.k.a. honeypot) per day, sometimes two.

I wouldn't object if this happened maybe once per week, but seeing how you can review at most 20 edits per day, getting 1-2 surveillance probes (i.e. up to 10%) within these seems a bit hilarious.

If review robots are really such a big problem, one needs to find a different solution. If community reviews really don't work, then maybe one should not have them in the first place (though I can hardly imagine that so many people run review bots, to what avail?).

In any case, the solution cannot be to harrass people who are actually trying to do review properly. I don't recall ever failing one of these, but still the system keeps bugging me. What for?

Doing a proper review involves not only skimming over the text and hitting "Reject!" on the first occasion. Or, not reading at all, and hitting "Accept!".
A proper review involves reading and trying to figure out what someone was trying to say, even if it looks wrong or is worded badly. Even it it comes over as gibberish -- not everyone is a native English speaker. Possibly an edit can be improved rather than rejected.

Reading the honeypot gibberish (which, to make it worse, sometimes looks almost like something a real person -- presumably a bit stoned -- could have written) takes time. It may come as a surprise, but time is valuable, even time that comes from a community "for free".

As it is now, I am getting more and more inclined to simply press "Reject!" on any edit that I see which doesn't look 100% right at the first glance. Because hey, it's probably another darn honeypot probe anyway. I'm probably not the first to feel like this, there are most likely people who already do just that.

Read as: Excessive honeypotting may drastically reduce the review quality.

At the very least, probe frequency could be reduced for people who have not failed any (or very few) probes in the past.

If someone has consistently passed a number of tests, either the tests cannot distinguish robots from humans, or the subject is really a human. In either case, there is no valid reason to persist doing them.

EDIT:
Since robo reviews are perceived as such a big problem, one might consider not giving out review rights automatically just because some reputation threshold is reached.

Instead, users could gain a "tentative review" privilegue. Meaning they can cast their approve/reject votes, but they do not necessarily approve or reject anything. That would mean more reviewers per edit are needed, of course (to to the real approves). As you have "correctly" approved so and so many edits (see Jack's "conformist tracking" suggestion below), you gain the real approve privilegue.

|improve this question
  • 1
    Perhaps the probability should exponentially decrease with each passed audit :) – Jack Mar 14 '13 at 9:35
  • 8
    Just for clarification - the Meta term of "robo reviews" refers to real people mindlessly clicking through every review they see, not actual automated review bots. – Pekka supports GoFundMonica Mar 14 '13 at 9:36
  • 3
    One a day is totally fine. One less review a day is really no big deal, there are more than enough reviewers around and the queues are always kept almost empty anyway. – Shadow The Princess Wizard Mar 14 '13 at 9:42
  • 1
    There is roughly 1 in 20, more if you review fast. – Toon Krijthe Mar 14 '13 at 9:49
  • 1
    My main point is that everyone (including me) is being "punished" for some (presumably few) people behaving inadequately. This has nothing to do with an "elite" either. The exponential decrease as suggested by @Jack would be an entirely acceptable solution. I'm not saying there should be no probes. I'm saying one should not pester people who aren't failing them and who are actually trying to do good reviews, because that discourages them from doing so. – Damon Mar 14 '13 at 9:51
  • 2
    Almost 90% of the audits are passed, but that also implies that more than 10% are not. I would not say that's "a few" and think it's a good idea to keep us on our toes. – Bart Mar 14 '13 at 9:56
  • @bart good info; 5-10% audits for a 10% fail rate seems about right. – Andrew Barber Mar 14 '13 at 10:10
  • For the record, I just had my 3rd today. – Damon Mar 14 '13 at 10:35
  • Damn... does the fact I found no honeypot yet mean I don't see them or that I'm lucky ? – Denys Séguret Mar 14 '13 at 10:40
14

I also see a lot of these. But I don't mind. It keeps me sharp, and eventually it will help solve the robo review problem (or at least make it managable).

I see no need for an elite force, that can be trusted. Besides, those honeypots are not harrassing us. It is just our daily sanity check.

|improve this answer
  • 2
    It does get a bit annoying after a while, especially if you are consistently passing the audits. Solving the robo reviewers problem is a priority, but we could adjust the frequency a bit for good reviewers, no reason to keep auditing them as frequently as those who failed their audits. – yannis Mar 14 '13 at 9:42
  • 1
    I have invented the game "spot the audit" if I see a review, I try to think if it is an audit or not. Right now I have a 70% succesrate. – Toon Krijthe Mar 14 '13 at 9:44
  • 2
    I play a variation of that game, I suspend people who make crappy edits, and pray the edit wasn't an audit. 0% success rate so far... – yannis Mar 14 '13 at 10:03
  • @Yannis, the universe is not fair. – Toon Krijthe Mar 14 '13 at 10:16
4

Firstly, the audits aren't just for robo reviewers. People reviewing in good faith might fat finger things by speeding through the queues (has happened to me as well once on a site without audits). The audits make sure that the robo reviewers get caught, and they make sure that "good" reviewers are reviewing slowly, reading through the posts.

Secondly, 1 a day isn't bad, if you're hitting the review cap every day. Reducing it would be counterproductive, it already is quite hard (and slow) to catch roboreviewers.

|improve this answer
  • Some of us get more, like 2-3 a day. After passing 100% for, say, 100 days straight, we just might consider reducing the number of audits. – Bo Persson Mar 14 '13 at 12:09
  • @BoPersson: Hmm, yeah, though it probably should be temporary (fail an audit, lose part of your credit rating). Point is, even the best folks can have days where they speed through the queue -- though that's a relatively minor issue. (Also, is 2-3 per day for just one queue? Or multiple?) – Manishearth Mar 14 '13 at 12:11
  • This is for the Proposed Edits queue alone. I'm probably a "fast reviewer" because I skip some obviously correct edits (leaving them for the robo reviewers) and code changes in languages I don't understand. – Bo Persson Mar 14 '13 at 12:20
1

Just throwing an idea in the ring here.

Because it's not very easy to find out the herd movement on a review, perhaps a better system to catch bad reviewers is by applying "conformist tracking".

For every reviewer, keep track of two statistics:

  1. Their response time within the last 24 hours, consistently fast response times would be suspect;

  2. Their response vs. other responses, e.g. "their view was shared by 0%, 20%, 40%, etc."

Unless 50%+ reviewers are bad, after a few reviews (you need a big enough sample size) it should become obvious who is gaming the system.

To combat against both mass rejects and mass approves, audits should have both negative and positive versions, i.e. audits that should be approved and those that should be rejected.

|improve this answer
  • "Their response vs. other responses" sounds like a good metric. Of course the sample size in a single edit is very small, but it would not hurt having 3-4 extra people cast a vote. And unless they are all (or the majority of them) robots, the grand total must eventually converge to "something sensible" (over a few dozen or so edits). If you're within maybe 90% of the crowd within that, you're OK. – Damon Mar 14 '13 at 10:18
  • @Damon I did mention "after a few reviews" :) of course, a single review is way not enough information. – Jack Mar 14 '13 at 10:20
  • 3
    You would think that the majority would be right, but often, they're not. I think it's because the don't-think approach is so much quicker. Often the first 3 repsonses to an edit are "approve" and those of us who took the time to read it get the dreaded "this edit has already been approved" message. It's not a normal vote where we all get our say. It's a race, and the ones DOING IT WRONG often win the race. – Kate Gregory Mar 14 '13 at 10:34
  • @KateGregory That basically asserts "50%+ reviewers are bad" ... that would be a shame indeed. – Jack Mar 14 '13 at 10:37
  • 1
    No, it asserts that out of a pool of reviewers large enough to contain 3 bad ones (lets say 20 for the heck of it), the 3 bad ones will all quickly make the wrong decision while the other 17 think about it. Once there are 3 reviews the decision is made. Slowing down bad reviewers might make them better, might make them go away, or might just allow enough time for Reject votes to come in from other reviewers. Me, I use Improve instead of Reject these days. It's slower and more work, and there's no chance to tell the suggester what was wrong, but at least it won't get approved. – Kate Gregory Mar 14 '13 at 10:46
  • @kate the race concept could still be captured by the first metric, i.e. response time. – Jack Mar 14 '13 at 11:09
  • It's a real problem that when the votes are Accept 3 - Reject 2, we can't assume that the reject votes are always wrong. It often happens to me that I try to add a third reject vote 2 seconds too late. – Bo Persson Mar 14 '13 at 12:14
  • @BoPersson: That's a design problem though, which would need to be addressed, instead of doing something else that is annoying and does not work either. There is no real reason why you couldn't be allowed to cast another vote even if 3 votes came in the second before, once you have a review item open, it's just designed badly. There is also no real reason why you couldn't be allowed to flag approvals that you perceive as obvious "didn't read, is robot" ones the same you can flag questions. This would probably work much better than probing every reviewer repeatedly. – Damon Mar 14 '13 at 15:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .