17

Recently I saw this edit to one of my posts:

https://stackoverflow.com/review/suggested-edits/4148594

It was proposed then accepted by three people in a matter of seconds. Refreshingly, another user ended up reverting the edit before I did.

I'm specifically calling out the reviewers; if you're on this list, try taking a half a second to think before pressing the button. The suggested edit queue isn't even close to being overloaded -- there's no excuse for rushing through them (and even if it was overloaded that's still no excuse).

  • johanandren has approved 22 edit suggestions and rejected 0 edit suggestions
  • Cheesebaron has approved 635 edit suggestions and rejected 268 edit suggestions
  • Len Jaffe has approved 24 edit suggestions and rejected 2 edit suggestions

I don't particularly care about the answer that was edited itself, it's just that I'm sick of seeing your crappy edit reviews. Please, shape it up.

It would be nice if we had a system that let other users flag edit reviews as crap, then permanently (or for a long time) took away the edit review privileges for people who were frequently flagged. The edit queue is totally manageable, we definitely don't need as many people with edit review privileges as we currently have.

Improve this question
5
  • 3
    Robo-reviewers!
    – Taryn
    Feb 24, 2014 at 18:14
  • Related: Shouldn't reviews be done a bit more carefully?
    – devnull
    Feb 24, 2014 at 18:14
  • 21
    robo reviewers don't read meta. You are preaching to the choir
    – Kate Gregory
    Feb 24, 2014 at 18:16
  • 3
    Is this just naming/shaming, or a suggestion to implement reviewer flags? I'm pretty sure the suggestion already exists as a feature request somewhere.
    – Geobits
    Feb 24, 2014 at 18:16
  • 9
    They're rushing through edit reviews because that queue isn't overloaded. You don't slow down when you see a flash of gold in the pan.
    – Bill the Lizard
    Feb 24, 2014 at 18:17

2 Answers 2

Reset to default
7

It would be nice if we had a system that let other users flag edit reviews as crap,

Yes, there exists a system.

If you feel that a suggested edit that shouldn't have been approved was actually approved, you can raise a custom flag describing the problem in detail. I've done that myself on several occasions (even requesting that the robo-reviewers be given a break). Not quite sure what happened to the robo-reviewers, but the flags were later found to be marked as helpful.

Improve this answer
4
  • 4
    Yeah, but which is more satisfying, flagging the post, or clicking a "This review sucks, and the reviewers should feel bad" button? Especially if Minitech was in charge of reviewing the results of said button?
    – LittleBobbyTables - Au Revoir
    Feb 24, 2014 at 18:23
  • @LBT I'd rollback the post and custom flag it. I wouldn't really mind adding up to the mod's queue. For me, it'd be very satisfying if the reviewers in question get a well-deserved break.
    – devnull
    Feb 24, 2014 at 18:26
  • 1
    Do you put the flag on the answer?
    – Jason C
    Feb 24, 2014 at 18:29
  • 1
    @JasonC If it's the question then on the question, else on the answer.
    – devnull
    Feb 24, 2014 at 18:30
13

Bill mentioned something that I've been mulling over for a while now...

We used to have a real problem with the suggested edit queue backing up, without enough active reviewers to handle the volume. So we dramatically reduced the requirements to be a reviewer, and that problem went away... But at the cost of some often sloppy reviewing.

We could turn back the dial a bit, raising the reputation needed to review or even requiring a certain number of past edits to have been made reviewers. The old criteria - 10K reputation - was too restrictive; there were entirely too many edits for the population of reviewers. But perhaps there's a happy medium to be had here somewhere...

Improve this answer
16
  • I would love to see the requirement raised a bit. It couldn't hurt to try it, worst case you lower it again until you find a good balance.
    – Jason C
    Feb 24, 2014 at 18:30
  • 2
    That would be fantastic. Maybe 7.5K if 10K is too restrictive.
    – devnull
    Feb 24, 2014 at 18:33
  • 10
    @JasonC The biggest problem is that evidence has shown a fairly week correlation between reputation and review quality. Even with a higher cap, you're often cutting out quality reviewers at the same proportion as lower quality reviewers. Ideally one would want to find some other criteria for restricting reviewers that has a stronger correlation to review quality.
    – Servy
    Feb 24, 2014 at 18:33
  • 3
    @Servy I couldn't agree more. Fundamentally, it makes no sense to have most of the privileges you get on this site be granted solely based on reputation. It's just a big rant that I didn't want to rehash here; the cons of the reputation-based system are sort of a given at this point. I've sometimes tossed around an idea for a separate "community moderator reputation" that drives privileges, but I haven't thought it through enough yet to make a solid proposal.
    – Jason C
    Feb 24, 2014 at 18:36
  • 1
    @JasonC It's actually been shown to be at least somewhat decent as a measurement of certain other site features, for example closing, which tends to be understood fairly well by users who've gotten 3k rep, even though a huge number of new users just don't get it. The main problem is that users who don't have the privilege aren't exposed to the feature. A 2k user doesn't have a good way of learning what edits should be approved. A 2k user is exposed to whether or not a question should be closed. So rep isn't a horrible privilege cutoff in genera, but here it is pretty bad.
    – Servy
    Feb 24, 2014 at 18:39
  • 1
    Maybe it would be beneficial if, say, once you reach the edit review reputation level, you have to go through (and pass a certain percentage of) those fake edit tests before you actually get the privilege (and perhaps, if you do not pass, you have to wait another few K of reputation before you get the chance again). Then the original rep level could remain low, and people with low rep are still given a chance to gain the privilege.
    – Jason C
    Feb 24, 2014 at 18:41
  • @Servy Maybe rep combined with other activity might be a good measure. Something akin to tag badges (m points, and n answers), so p rep and something to measure other activity (# of posts, revisions, something else...).
    – devnull
    Feb 24, 2014 at 18:44
  • @JasonC Been suggested before. Personally I don't see it being effective. The audits, as they stand, are there to make sure that the reader is reading the post. No more than that. If you read it, and no nothing else about whether or not it's an appropriate edit, you ought to pass. Users should really have a higher standard than "I read the post" to be an acceptable reviewer. There are of course other problems with them as well.
    – Servy
    Feb 24, 2014 at 18:47
  • @devnull Personally I don't see how those would have a stronger correlation to review quality than just rep. You could look into it, sure, but conceptually those are measuring how good someone is at answering, not how good someone is at either editing, or reviewing edits.
    – Servy
    Feb 24, 2014 at 18:49
  • @Shog9 BTW, I do stand by raising the bar, but as a case against that I still want to acknowledge that the reviewer who appropriately rolled back the edit I mentioned had 5400 rep, which would be below a high cutoff.
    – Jason C
    Feb 24, 2014 at 18:49
  • @Servy But, do you think that if users were forced to go through enough audits (say, 20-30) before getting the review privilege (and you had to pass a certain percentage), it would give people enough exposure to get off on the right foot?
    – Jason C
    Feb 24, 2014 at 18:52
  • 3
    @JasonC No, I don't. Audits are all cases of obvious and blatant vandalism. In fact, someone could pass a series of 30 of the current audits by pressing "reject" every single time without even reading them. There are no good audits, there are no questionable cases, there are no instances of the types of issues that many people mistakenly think are fine, but that in fact are violating review standards. The entire auditing system would need to be entirely revamped, in very dramatic ways, for it to be an effective entry exam.
    – Servy
    Feb 24, 2014 at 18:57
  • @JasonC Since you're really pursuing this, just take a look at this question. Particularly the comments to my answer address your exact proposal in pretty significant depth.
    – Servy
    Feb 24, 2014 at 18:58
  • 1
    @Jason: yeah, I'm aware that reputation has a fairly weak connection to reviewer acumen. The purpose of raising the threshold would be to simply ensure there were more reviews pending most of the time - not a lot, but enough to hopefully reduce the temptation to blast through as quickly as possible to "defend your claim".
    – Shog9
    Feb 24, 2014 at 19:47
  • 1
    They're two separate systems, @Lower. I've spent years looking at and helping to tweak both of them when I'm able. Keep in mind, the close queue has been slowly growing for two and a half years now, during which time suggested edits went from being a novel curiosity to having their own crushing backlog to the current state of affairs. There's no silver bullet; just a lot of little steps forward and back... So let's try to stay focused without losing perspective.
    – Shog9
    Feb 25, 2014 at 3:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .