80

There has been a lot of discussion/complaints about the quality of reviews. Many have noted that a large (maybe even a majority) of reviews are rubber stamps in pursuit of a Steward badge (especially because it's one of the few gold badges that can be attained without monumental effort if you're willing to do a crappy job). Many suggestions have been thrown out there - some good and some bad - but nothing that everyone can agree to:

  • Penalize (rep or suspend) reviewers who disagree with the majority more than X times - Terrible idea in my option, as the majority is often wrong, many reviews are legitimately subjective, and it punishes prolific reviewers regardless of quality.

  • Penalize (rep or suspend) reviewers who disagree with the majority more than Y% of the time - Worthy suggestion, but suffers most of the problems as above

  • Allow reviews to be flag-able - This is a great idea, but without a review-of-review queue flags won't happen nearly enough to keep up with the huge volumes of reviews... it's basically just luck that certain bad reviews will get caught, and if somehow users were doing a good job for catching the massing number of bad reviews, it's not clear that moderators could keep up (and flags are generally reserved for egregious issues)

  • Create a review of reviews - I personally like this idea, but others seem concerned (legitimately) that it would suffer the same rubber-stamp abuse as the other review queues and therefore raise the question of needing a "review of reviews or reviews...". I still think it would help, but it is an issue.

  • Raise the Rep cap required to review or require an invite-only review process (i.e. "stop giving every monkey with 2K instant access to /review" - I think this might help, but a lot of bad reviews come from high-rep users, and high-rep users are sometimes high rep because they are reward-seekers (i.e. cherry pick simple questions, engage in puppet voting, etc) and are the most likely to abuse the system. Many high rep users are also just early adopters who answered or asked a few simple posts that have garnered hundreds of votes because they were the first ones to post that issue on SO. Bottom line: rep != quality, though again, it would probably help a little. Having an invite-only review process would help even more, but it might limit the pool too much.

  • Remove the review badge incentive - This has been proposed a couple of times and would definitely help solve the problem. However, the badges were introduced for a reason: not enough reviews were occurring, and we have to balance the need to prevent bad reviews with the need to induce users to actually help with the review queue.

  • Sort the review queue so reviewers can only review posts with tags matching their rep profile (i.e. I couldn't review posts about C# because I have virtually no rep associated with the C# tag) - This is a great idea in my eyes, but I don't think it would solve the problem. Many (maybe most) of the bad reviews don't seem to stem from ignorance of the subject, but from lazy/rubber stamp reviewing (no attempt to understand the post or its context). Also, many reviews require no technical expertise, so some good eyes wouldn't get to see the posts. Overall I think it would help and I endorse this, but it wouldn't dramatically impact the bad-review phenomenon.

  • Use time-based algorithms to make sure people aren't speeding through reviews - We should have these features, but often all you need is 5 seconds (link-only answers, incomprehensible one-word posts, obvious duplicates, etc). There should be an algorithm to help detect bad users, but it should be subject to human oversight.

  • Make it easier to see the context of posts being reviewed (see associated answers, etc without forcing reviewers to open up separate pages) - I'm a big fan of this one, but it only helps reviewers who are genuinely trying to be good actors... bad reviewers will keep plugging along as normal.

  • Increase the number of votes required to approve/close etc - This has already been partially implemented and is a good idea, but still runs into issues of rubber-stamping, and it's been my (unsubstantiated) suspicion that in the close vote queue the more close votes an item already has the more likely it is to be closed by subsequent users who are banking on previous reviews having done the due-dilligence (I'd love to see numbers on that).

  • Make the "Looks Good"/Skip buttons increment the review count - Would definitely help reduce bad reviews, but encourages badge-seekers to just switch from approve/upvote to skip (hopefully). This undermines the purpose of offering review incentives, though I agree that it's better to have the folks getting badges for null reviews than bad reviews.

There are definitely more suggestions (good and bad), but so far most solutions see to either only nudge reviewing in the right direction or relies on there being enough good actors that algorithms will accurately detect bad actors - which I strongly suspect isn't the case - or don't factor in the legitimate ambiguity/subjectivity of many reviews.

This is hardly a replacement for these suggestions, but here's a new one to throw onto the pile: Manually create "Honeypot" items in the review queues

These items are purposely created bad edits/bad late answers/bad close votes/etc - they are created so that they are either objectively wrong or (for close votes) so clearly within or outside the bounds of the FAQ that no reasonable reviewer could disagree on what the correct vote would be. When a user enters the review queue they would be warned that such questions exist and that were are penalties (removal of review badges, loss of rep, temporary suspension of review privileges... I'm not sure what's best) for voting "wrong" when they happen to encounter one of these questions (I think users should get a pass for misjudging a tiny percentage of these, as I've accidentally hit the wrong button a couple of times).

I do think that penalties should be appealable on the basis of a long and positive review history, but I think that just the knowledge that there are "Objectively wrong answers" will keep reviewers much more honest and diligent, even if only a small percent of the review queue is composed of honeypot posts. It would also encourage more use of the Skip/Not sure button when users aren't sure about an answer.

There are some obvious concerns:

  • disincentivising reviewing - I agree, but there are good incentives in place and the community is currently flying through the review queue... this will slow down a little bit, but it should disproportionately exclude reviewers who are currently gaming the system.

  • Honeypot items would have to come from select trusted users (mods etc) and would require time to craft - This is a big one, but surmountable I think (mods: correct me if I'm wrong). The number of honeypot posts could be quite small and you could recycle them with some frequency (maybe altering the text slightly).

  • Would change the culture of SO/Offends users/presumes users are suspicious: This is true, and perhaps it's more overtly distrusting than restrictions based on rep or requiring multiple reviews to confirm, but I think that there is enough recognition on the part of good reviewers that there is a problem with the review process that they'd tolerate it.

Anyway, this may be a terrible idea, but I do think something fairly drastic needs to be done, as it seems that a lot of the most diligent users are being driven away from the review process by the reward-seeking behavior of so a large percent of reviewers.

21
  • 17
    RE: "Honeypot items would have to come from select trusted users (mods etc)..." You could start with posts that mods and other trusted users have already reviewed. Oct 26, 2012 at 18:25
  • 1
    @BilltheLizard - I like that. I think it would have to be specially marked (i.e. not only should it be accepted/rejected, but it's also not ambiguous/subjective)
    – Ben D
    Oct 26, 2012 at 18:28
  • 8
    Isn't this status-completed? This has been around since the start of the review beta, although currently there are no negative consequences for messing up on these reviews (yet; I assume that's to be implemented pending analysis of the results).
    – Servy
    Oct 26, 2012 at 18:29
  • 2
    Why are you analyzing a large number of previous suggesting in this question? How are any of them relevant to your suggestion (other than to say that there are other suggestions). It seems like that entire section of the question could just be deleted.
    – Servy
    Oct 26, 2012 at 18:31
  • @Servy is there a link for that? I'm not second-guessing you, I just haven't seen it brought up in other threads on this subject. If it exists and it's not common knowledge it can't do its job of dissuading bad reviews.
    – Ben D
    Oct 26, 2012 at 18:32
  • @BenD I'm pretty sure it's not applicable to first/late posts, as there are too many possible options. It's only in low quality posts (and maybe close votes?). In any case, currently when you find such a question, if you choose the wrong option (i.e. looks good an obviously bad post) it brings you to a screen saying, "actually...this post is [...]" and you need to confirm that you understand why you were wrong. Beyond this mildly insulting dialog, and having the results logged for potential future use, nothing else happens. The idea is to help the team learn from the recorded results.
    – Servy
    Oct 26, 2012 at 18:35
  • 1
    @Servy - Mostly because I'm acknowledging that this is one of many things that can/should be done to reviews... not a replacement for other suggestions. It also helps contextualize why I this suggestion would complement/not be superseded by existing suggestions
    – Ben D
    Oct 26, 2012 at 18:35
  • @BenD I think you could have said that in a sentence or two, like you just did in that comment, rather than in over a screen's worth of text. You also should probably put the actual suggestion at the top, at the very least, so that people don't need to dig to find what the actual suggestion is.
    – Servy
    Oct 26, 2012 at 18:37
  • @Servy - fair enough, but I'm trying to avoid answers like "Why not do [x]?"... I'm probably being overly verbose, but I'm attempting to show that I've given this some real thought and that I'm aware of other suggested solutions.
    – Ben D
    Oct 26, 2012 at 18:39
  • @BenD Instead your lack of research is just demonstrated by suggesting something that was implemented months ago...
    – Servy
    Oct 26, 2012 at 18:41
  • @Servy Again: could you provide a link or some documentation that this has been implemented? I'm happy to remove to change the question (perhaps to "why hasn't this been implemented?"), but I looked and found nothing similar on meta.
    – Ben D
    Oct 26, 2012 at 18:53
  • 1
    @BenD meta.stackexchange.com/questions/142998/…
    – Servy
    Oct 26, 2012 at 18:56
  • @Servy - Thanks! Just seconded by Shog9.
    – Ben D
    Oct 26, 2012 at 18:58
  • 3
    @Servy could you include some kind of description with links in the future? It's annoying to have to click on a naked link to find out where it points. (That's exactly the answer I wanted to link to when I saw the word "honeypot," in fact.)
    – Pops
    Oct 26, 2012 at 19:01
  • 6
    I don't care whether some details of this are already implemented or not: it's still a great summary of the current situation and the problems with the queue.
    – Pekka
    Oct 26, 2012 at 21:05

3 Answers 3

46
+150

Actually, we're already doing some of this - if you spend some time in the Low Quality queue, you'll almost certainly encounter a "honeypot" post sooner or later - one that's been overwhelmingly judged by the community as high or low quality. If you respond incorrectly, you get a warning - and the failure is logged. We're not currently using this for anything other than education, but kicking you out of the queue for the day in response to multiple failures would be a pretty safe and effective response I think.

Given some of the (legitimate) complaints about careless voting in the First Post and Late Answer queues, I'd like to see this model expanded to that as well. [Edit: this is now in place for First Posts, to be rolled out to Late Answers in the near future. ] Now, voting is a pretty subjective affair - but I think it's a safe bet that up-voting a question posted as an answer (or nonsense / spam) is a pretty good indication of someone who's not paying attention. Particularly if they spent less than 10 seconds "reviewing" the post.

I particularly like this as a tool for education, an opportunity to instruct reviewers in the behavior we expect from them.

The model for a lot of this - wikiHow's Community Dashboard - goes even further, automatically rolling back your changes in response to persistent abuse. That's also a possibility for some actions, but would be hard to pull off for others.

3
  • 13
    Rollbacks are something we could use for the Suggested Edits queue, though. Right now folks are temporarily banned from suggesting edits if a few are rejected in a short amount of time. I bet we could extend that to blocking people from reviewing (and/or editing) for a time if a number of edits they approved get rolled back in a given time period.
    – Adam Lear StaffMod
    Oct 26, 2012 at 19:04
  • 4
    This is great to know. I'd love to see a more aggressive implementation, as well as a clear warning/note that honeypot questions exist I suspect that just the knowledge that you could be "caught in the act" would make some people more thoughtful.
    – Ben D
    Oct 26, 2012 at 19:07
  • 2
    I like this idea, but I have found in the past a couple of instances (not many, but I do try to do my due diligence) where I disagreed with the Honepot low quality post. Though as long as it isn't a "one-strike" thing, it should be fine.
    – Jack
    Nov 13, 2012 at 22:26
22

Update 2013-01-29 - Suggested Edits queue

Suggested Edits queue has been expanded with "honeypot" items, as announced here:

We've rolled out audit tasks for suggested edits in our more recent builds.

...Early numbers suggests double-digit percentages of reviewers don't reliably reject these nonsense edits...


Update 2012-11-27 - First Posts queue

First Posts queue has been expanded with "honeypot" items. Particular example (link) answer (blatant , to be precise) shown to me was like:

I have same problem now. <blah-blah> What can I do? Please help. <blah-blah>

Right after I downvoted, a message popped up:

Congratulations!

This was only a test, designed to make sure you were paying attention. Down-voting, flagging this as "very low quality", or voting to close this post are all appropriate actions in this case.

https://i.stack.imgur.com/WbSCi.png


Inspired by above, I figured that "honeypot" for robo-approvers in the Suggested Edits queue would be also quite easy to set up (follow-up - it's there at last, see Update 2013-01-29).

Honeypots in Close Votes queue can be planted as suggestions for blatantly wrong duplicates...

Late Answers queue appears to have honeypots / test items as is, without any additional changes...

Oh my this is all doable. What are we waiting for?

17
  • 2
    +1 Me to :)
    – Himanshu
    Nov 27, 2012 at 10:20
  • 3
    I have seen a couple of them too. Some are quite obvious: an answer to a deleted question. But is sure beats mindless upvotes. Nov 27, 2012 at 12:53
  • For the suggested edits: take a post with several problems and correct only one or two minor ones is also a good one. Nov 27, 2012 at 12:54
  • 2
    @GamecatisToonKrijthe edit "several" into "severe" in your comment while in grace period, and that would be a perfect heuristics, too :)
    – gnat
    Nov 27, 2012 at 12:56
  • 2
    Another supply of honeypots for close votes could be "off-topic" for rejected migrations.
    – McCannot
    Nov 27, 2012 at 15:48
  • 1
    Might wanna keep an eye on suggested edits... There's a real treat in there if you get "lucky".
    – Shog9
    Jan 29, 2013 at 3:48
  • @Shog9 wow I really missed this change. Been focused rather heavily on hotness formula, could not afford doing difficult reviews... and suggested edits are difficult to me, I skip quite a lot in this queue
    – gnat
    Jan 30, 2013 at 6:55
  • update there seem to be also Reopen Question review audit introduced recently
    – gnat
    Jan 31, 2013 at 14:16
  • @Shog9 - any news on when users are going to be suspended for failing Suggested Edit audits? There's one user in particular I'm thinking of who has failed their last 8 straight audits, with a 442/14 approve/reject rate. That can't be good. Feb 11, 2013 at 16:12
  • 2
    Users are blocked from /review for a period of time, not suspended, after failing a sufficient number of audits, @LittleBobbyTables. This is enabled for suggested edits, however we'll be tightening up the criteria a bit this week since some obviously careless reviewers are getting missed.
    – Shog9
    Feb 11, 2013 at 17:46
  • Is there a limit where you stop getting honeypot questions? I'm 3 flags shy of 700 flags on ServerFault and I'm still getting honeypot questions. I'm fairly sure I know what I'm doing at this point.
    – tombull89
    May 15, 2013 at 20:05
  • @tombull89 unlikely: I have about twice as much flags at Programmers (~1300) and I am getting honeypots. BTW I am also still failing some of them, as well as making mistakes in flagging - which is probably the reason why audits won't stop (no matter how good one is now, there's no guarantee this will be always so in the future)
    – gnat
    May 15, 2013 at 21:53
  • @gnat okay - is there a way to see failures apart from when you are in the review queue? I can't remember last time I failed an audit.
    – tombull89
    May 16, 2013 at 7:38
  • @tombull89 in theory, all the reviews, including failed, as well as passed audits are expected to be logged in your recent-activity profile section, in the sub-tab reviews
    – gnat
    May 16, 2013 at 7:49
  • @gnat thanks - although your first link does not link to where I think it should have gone. And "in theory?"
    – tombull89
    May 16, 2013 at 7:51
3

I see some people frequenting some review queues for weeks until they reach the magical 1000. Some add a few extra reviews but then they are gone for that queue.

Of course, if they are bad reviewers, that is not a disaster, but I think that behaviour is suspicious too.

1
  • 1
    Honestly, I don't think this behavior suspicious. There are different people in the community and for some of them the badges matters. As long as they review correctly I don't mind if they stop at 1000. For my part, I did it mostly to learn about SO and improve my own answering. I also like to earn badges and I think it shows a certain involvement in the community. However, after reviewing more than a thousand First Post, I became bored to review those type because the patterns are often similar and are redundant. Same for the Late Answer which ends up not being a real answer most of the time.
    – ForceMagic
    Feb 6, 2014 at 17:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .