4

This seems to occur on all Stack Exchange sites and I'm not going to provide any specific examples as that is basically naming and shaming. However, as many of you who work the review queues know, and maybe even some people who submit content to the review queues to be reviewed; the quality and consistency of some reviewers is substandard.

Some general examples:

  • Edits that improve posts being rejected as "an attempt to reply to the post"
  • Reviewers that have approved 100% of all suggested edits they've ever reviewed, even when others have tried to reject them (ie: they just keep clicking Accept for everything)
  • Reviewers who click "No Action Required" for spam and off topic posts

Is there anything that could be implemented so that even established users get audited every now and again, with punishments for failing the audit being remembered and escalating if they keep failing future audits (up to and including perhaps revoking the privilege to work that queue)?

This isn't about one off mistakes, this is about the people who see the review queues as another source of badges and consistently apply the same action to everything just to increase a number and move a progress bar.

I feel that letting this continue as it is results in the duplication of effort on the part of the reviewers who actually pay attention when working through the queues by approving changes that shouldn't be approved, by ignoring content that needs fixing, and allowing more negative posts like spam slip through the net and stick around for longer than it had to.

Can we do something to reduce the amount of poor quality reviewers there are working the queues?

  • 3
    Audits on all SE sites! – user98085 Dec 18 '13 at 15:14
  • 1
    (That was not a serious request, audits are mean and even 'proper' reviewers fail them a little too often.) – user98085 Dec 18 '13 at 15:16
  • 4
    I think this is a duplicate of meta.stackexchange.com/q/152709/164138? – THelper Dec 18 '13 at 15:18
  • There may be some poor quality reviews, but look at the size of the review queue! We are almost below 100k again. – Travis J Dec 18 '13 at 15:18
  • @FEichinger: I've mis-clicked. I actually wanted to choose the question that THelper mentioned. – ProgramFOX Dec 18 '13 at 15:20
  • @ProgramFOX Alrighty, yeh, that makes sense. – user98085 Dec 18 '13 at 15:21
  • 11
    I spent 15 minutes yesterday editing a horrible question so I could reopen it. When I finished, it turned out five reviewers (all non-native English speakers, I believe) had already reopened it. There were only 2 questions in the reopen queue at the time. I think a key piece of the problem is that people skim through quickly if they fear somebody else is going to review it first, rendering their work moot. And if that's so, we could improve the situation by not letting multiple people review an item simultaneously (subject to conditions, of course). – mmyers Dec 18 '13 at 15:23
  • 1
    Perhaps if you edit something in the queue you get to check it out, or some other metric which prevents it being reviewed while editing. – Travis J Dec 18 '13 at 15:24
  • @mmyers again without providing examples (which I could provide for SO right now, but it would basically be naming and shaming) - there are instances where this isn't even the case and the user(s) in question is/are just clicking "approve" on everything – Flyk Dec 18 '13 at 15:25
  • @Flyk: Blindly clicking "approve" on everything can be solved: even if the reviewer never fails an audit, they can still be manually banned from reviewing by a moderator. – mmyers Dec 18 '13 at 15:31
  • "an attempt to reply to the post" is really "invalid or an attempt to reply to the post" - I use that reject reason often when the edit is just "invalid". – laalto Dec 18 '13 at 15:40
  • @mmyers "I think a key piece of the problem is that people skim through quickly if they fear somebody else is going to review it first, rendering their work moot. And if that's so, we could improve the situation by not letting multiple people review an item simultaneously" -- you just identified exactly why most SO answers suck, and how to fix it: only let one person answer a question at a time – Michael Mrozek Dec 18 '13 at 16:06
10

For your general examples:

  1. Many of the minor edits that are rejected by members of the community aren't rejected because the reviewers are gaming badges, they're done so because these people don't like the edits. That's an important distinction. There's still quite a bit of debate in that grey area as to what's too minor or what kind of corrective edits should be allowed in the technical content of answers. Until everyone agrees that something is clearly wrong or right, there's no way to even design a test for this.

  2. No one can approve 100% of the suggested edits they see, beyond a certain number of edits. The suggested edit audits ensure that. Those are always rejection cases, and they do work very well from what I've seen. The audits have hugely reduced the number of people gaming suggested edit reviews.

  3. This is the area of greatest concern for me currently, when it comes to reviews. Given that audits exist for both the approval and rejection cases on First Posts and Late Answers, I have observed some people auto-approve everything (or blindly upvote or downvote everything) and make it a ways before they get caught by the audits. Requiring a second review for elements in this queue usually seems to prevent spam from getting through due to these reviewers, but sometimes it doesn't.

Moderators have access to tools that allow us to look into the review history of a user (although that could be made a lot more convenient), and a few of us regularly watch for anomalous patterns in the review queues. Personally, I like looking at the review history of spam that no one caught, and if I find someone that has approved it, I will take a close look at their review history. Usually, they end up taking a vacation from review as a result.

Of the three examples you provide, only abuse in the Late Answers and First Posts queue seems to be a significant problem anymore. Other bad reviews that I've seen in suggested edits, etc. don't appear to be part of a pattern for a lot of users, just honest mistakes or true differences of opinion. I am a little concerned with the tendency for people in the Close Votes queue to default to closing everything there, and have seen some good questions blindly closed, but again that usually isn't part of a trend of bad reviews.

  • 3
    My post was more focusing on the approving of everything, with a bit of rejecting for the wrong reason thrown in because it's related. Regarding (2), I can find you a user quite easily who has nearly 300 approved edits with no rejections, along with a couple of approvals that should have been rejections (including one that edits otherwise correctly spelt text into a sentence full of typos) on SO but I specifically wanted to avoid turning this into a name and shame session – Flyk Dec 18 '13 at 15:57
  • 1
    @Flyk - Flag that user or one of their posts with a custom flag. I'd like to take a look at that, because a 0% suggested edit rejection rate shouldn't be possible under the current system on SO. People occasionally find exploits, and we like to know about them. That also applies to other users who have obvious patterns of review abuse, because we'll look into them. – Brad Larson Dec 18 '13 at 16:07
  • 1
    done with an "FAO Brad Larson" on it – Flyk Dec 18 '13 at 16:09
  • 1
    @Flyk - Ah, I see. Yeah, there's something odd there. They've never passed an audit, and should have been banned with the latest failure. I've taken care of that, but I'll pass it along for someone to look at in more depth. They shouldn't have been allowed to get this far. Thanks for the flag. – Brad Larson Dec 18 '13 at 16:26
  • Thanks - was the initial cause for the question since obviously the current audit system is not without holes that allow things like this to slip through – Flyk Dec 18 '13 at 16:30

Not the answer you're looking for? Browse other questions tagged .