Improving the Quality of Reviews: Project Honeypot

Question

There has been a lot of discussion/complaints about the quality of reviews. Many have noted that a large (maybe even a majority) of reviews are rubber stamps in pursuit of a Steward badge (especially because it's one of the few gold badges that can be attained without monumental effort if you're willing to do a crappy job). Many suggestions have been thrown out there - some good and some bad - but nothing that everyone can agree to:

• Penalize (rep or suspend) reviewers who disagree with the majority more than X times - Terrible idea in my option, as the majority is often wrong, many reviews are legitimately subjective, and it punishes prolific reviewers regardless of quality.

• Penalize (rep or suspend) reviewers who disagree with the majority more than Y% of the time - Worthy suggestion, but suffers most of the problems as above

• Allow reviews to be flag-able - This is a great idea, but without a review-of-review queue flags won't happen nearly enough to keep up with the huge volumes of reviews... it's basically just luck that certain bad reviews will get caught, and if somehow users were doing a good job for catching the massing number of bad reviews, it's not clear that moderators could keep up (and flags are generally reserved for egregious issues)

• Create a review of reviews - I personally like this idea, but others seem concerned (legitimately) that it would suffer the same rubber-stamp abuse as the other review queues and therefore raise the question of needing a "review of reviews or reviews...". I still think it would help, but it is an issue.

• Raise the Rep cap required to review or require an invite-only review process (i.e. "stop giving every monkey with 2K instant access to /review" - I think this might help, but a lot of bad reviews come from high-rep users, and high-rep users are sometimes high rep because they are reward-seekers (i.e. cherry pick simple questions, engage in puppet voting, etc) and are the most likely to abuse the system. Many high rep users are also just early adopters who answered or asked a few simple posts that have garnered hundreds of votes because they were the first ones to post that issue on SO. Bottom line: rep != quality, though again, it would probably help a little. Having an invite-only review process would help even more, but it might limit the pool too much.

• Remove the review badge incentive - This has been proposed a couple of times and would definitely help solve the problem. However, the badges were introduced for a reason: not enough reviews were occurring, and we have to balance the need to prevent bad reviews with the need to induce users to actually help with the review queue.

• Sort the review queue so reviewers can only review posts with tags matching their rep profile (i.e. I couldn't review posts about C# because I have virtually no rep associated with the C# tag) - This is a great idea in my eyes, but I don't think it would solve the problem. Many (maybe most) of the bad reviews don't seem to stem from ignorance of the subject, but from lazy/rubber stamp reviewing (no attempt to understand the post or its context). Also, many reviews require no technical expertise, so some good eyes wouldn't get to see the posts. Overall I think it would help and I endorse this, but it wouldn't dramatically impact the bad-review phenomenon.

• Use time-based algorithms to make sure people aren't speeding through reviews - We should have these features, but often all you need is 5 seconds (link-only answers, incomprehensible one-word posts, obvious duplicates, etc). There should be an algorithm to help detect bad users, but it should be subject to human oversight.

• Make it easier to see the context of posts being reviewed (see associated answers, etc without forcing reviewers to open up separate pages) - I'm a big fan of this one, but it only helps reviewers who are genuinely trying to be good actors... bad reviewers will keep plugging along as normal.

• Increase the number of votes required to approve/close etc - This has already been partially implemented and is a good idea, but still runs into issues of rubber-stamping, and it's been my (unsubstantiated) suspicion that in the close vote queue the more close votes an item already has the more likely it is to be closed by subsequent users who are banking on previous reviews having done the due-dilligence (I'd love to see numbers on that).

• Make the "Looks Good"/Skip buttons increment the review count - Would definitely help reduce bad reviews, but encourages badge-seekers to just switch from approve/upvote to skip (hopefully). This undermines the purpose of offering review incentives, though I agree that it's better to have the folks getting badges for null reviews than bad reviews.

There are definitely more suggestions (good and bad), but so far most solutions see to either only nudge reviewing in the right direction or relies on there being enough good actors that algorithms will accurately detect bad actors - which I strongly suspect isn't the case - or don't factor in the legitimate ambiguity/subjectivity of many reviews.

This is hardly a replacement for these suggestions, but here's a new one to throw onto the pile: Manually create "Honeypot" items in the review queues

These items are purposely created bad edits/bad late answers/bad close votes/etc - they are created so that they are either objectively wrong or (for close votes) so clearly within or outside the bounds of the FAQ that no reasonable reviewer could disagree on what the correct vote would be. When a user enters the review queue they would be warned that such questions exist and that were are penalties (removal of review badges, loss of rep, temporary suspension of review privileges... I'm not sure what's best) for voting "wrong" when they happen to encounter one of these questions (I think users should get a pass for misjudging a tiny percentage of these, as I've accidentally hit the wrong button a couple of times).

I do think that penalties should be appealable on the basis of a long and positive review history, but I think that just the knowledge that there are "Objectively wrong answers" will keep reviewers much more honest and diligent, even if only a small percent of the review queue is composed of honeypot posts. It would also encourage more use of the Skip/Not sure button when users aren't sure about an answer.

There are some obvious concerns:

• disincentivising reviewing - I agree, but there are good incentives in place and the community is currently flying through the review queue... this will slow down a little bit, but it should disproportionately exclude reviewers who are currently gaming the system.

• Honeypot items would have to come from select trusted users (mods etc) and would require time to craft - This is a big one, but surmountable I think (mods: correct me if I'm wrong). The number of honeypot posts could be quite small and you could recycle them with some frequency (maybe altering the text slightly).

• Would change the culture of SO/Offends users/presumes users are suspicious: This is true, and perhaps it's more overtly distrusting than restrictions based on rep or requiring multiple reviews to confirm, but I think that there is enough recognition on the part of good reviewers that there is a problem with the review process that they'd tolerate it.

Anyway, this may be a terrible idea, but I do think something fairly drastic needs to be done, as it seems that a lot of the most diligent users are being driven away from the review process by the reward-seeking behavior of so a large percent of reviewers.

Answer 1

Actually, we're already doing some of this - if you spend some time in the Low Quality queue, you'll almost certainly encounter a "honeypot" post sooner or later - one that's been overwhelmingly judged by the community as high or low quality. If you respond incorrectly, you get a warning - and the failure is logged. We're not currently using this for anything other than education, but kicking you out of the queue for the day in response to multiple failures would be a pretty safe and effective response I think.

Given some of the (legitimate) complaints about careless voting in the First Post and Late Answer queues, I'd like to see this model expanded to that as well. [Edit: this is now in place for First Posts, to be rolled out to Late Answers in the near future. ] Now, voting is a pretty subjective affair - but I think it's a safe bet that up-voting a question posted as an answer (or nonsense / spam) is a pretty good indication of someone who's not paying attention. Particularly if they spent less than 10 seconds "reviewing" the post.

I particularly like this as a tool for education, an opportunity to instruct reviewers in the behavior we expect from them.

The model for a lot of this - wikiHow's Community Dashboard - goes even further, automatically rolling back your changes in response to persistent abuse. That's also a possibility for some actions, but would be hard to pull off for others.

Answer 2

Update 2013-01-29 - Suggested Edits queue

Suggested Edits queue has been expanded with "honeypot" items, as announced here:

We've rolled out audit tasks for suggested edits in our more recent builds.

...Early numbers suggests double-digit percentages of reviewers don't reliably reject these nonsense edits...

---

Update 2012-11-27 - First Posts queue

First Posts queue has been expanded with "honeypot" items. Particular example ^(link) answer (blatant not-an-answer, to be precise) shown to me was like:

I have same problem now. <blah-blah> What can I do? Please help. <blah-blah>

Right after I downvoted, a message popped up:

Congratulations!

This was only a test, designed to make sure you were paying attention. Down-voting, flagging this as "very low quality", or voting to close this post are all appropriate actions in this case.

---

Inspired by above, I figured that "honeypot" for robo-approvers in the Suggested Edits queue would be also quite easy to set up (follow-up - it's there at last, see Update 2013-01-29).

• Just take a reasonably good post, make a suggestion to wrap it into senseless "Hi" and "Thanks", add an absurd edit comment like "improved formatting" and voila. Handful reviews proven fake => two weeks review suspension.

Honeypots in Close Votes queue can be planted as suggestions for blatantly wrong duplicates...

Late Answers queue appears to have honeypots / test items as is, without any additional changes...

Oh my this is all doable. What are we waiting for?

Answer 3

I see some people frequenting some review queues for weeks until they reach the magical 1000. Some add a few extra reviews but then they are gone for that queue.

Of course, if they are bad reviewers, that is not a disaster, but I think that behaviour is suspicious too.