52

Currently, the threshold for approving or rejecting a suggested edit is two approve / reject votes, whichever is reached first.

This means that, when bad or just dubious edits get approved, it's often with a vote count of something like 1 reject / 2 approve (or 2 reject / 3 approve on SO). Typically, these edits manage to get one or two very quick approve votes from, um, fast and inattentive reviewers, after which it's very hard for the slower and more careful reviewers to accumulate enough votes to catch up before the edit is approved.

Conversely, when a valid edit gets wrongly rejected, this very rarely happens unanimously either. More often, you'll see something like 3 votes to reject / 2 to approve, but that single last vote, no matter how well or poorly considered, is enough to tip the scales.

So I'd like to propose a simple change to the approval / rejection threshold:

  • Suggested edits should be approved when S = (# of approve votes − # or reject votes) ≥ N, where N = 3 on SO and N = 2 on other sites.

  • Similarly, suggested edits should be rejected when S ≤ −N.

No, this change is not going to magically fix all the problems with suggested edit review. But I believe that it would be a simple and conservative improvement over the current behavior:

  • For obviously good or bad edits that get approved / rejected unanimously, this suggestion would not change the current behavior in any way.

  • For edits that get one mistaken or careless reject / approve vote (as happens pretty often), an extra approve / reject vote would be required to balance it.

    This would slightly slow the review process for such edits, but not excessively so. Further, I would argue that these are precisely the kind of edits that deserve that little bit of extra attention to make sure the eventual decision will be correct.

  • Edits that are truly controversial, and for which the reviewer community is nearly exactly 50–50 split between voting to approve and reject, might stay in the review queue significantly longer (maybe twice as long on average) than under the current system. However, this is a trade-off for the increased likelihood that the eventual decision will actually reflect the majority opinion among reviewers, rather than mere random chance.

    In any case, even if reviewers are exactly 50–50 split, the expected number of votes needed to reach S = ±N by random chance under the proposed scheme is only N², i.e. 4 votes for other sites and 9 votes for SO. Even a slight bias in the voting will further reduce this. Besides, the longer the suggestion remains in the queue, the more likely it gets that someone will short-circuit the process by casting a binding vote (mod / post owner) or by choosing to improve the edit.


Ps. If it is felt that there's an issue with approval being easier than rejection, a simple further chance to address that could be to make each reject vote count as, say, −2 rather than just −1 approval votes (and change the rejection threshold to S ≤ −2 × N to compensate).

This still wouldn't help with bad edits getting unanimously approved (which isn't really something that can be fixed, except maybe by slugster's cattle prod), but it would mean that each reject vote would now effectively cancel out two approve votes, titlting the balance for disputed edit suggestions in favor of rejection and making it easier for more careful reviewers to counter the "robo-approvers".

  • Not convinced. I see many bad edits get three-nil Approve (whilst I'm checking, then Rejecting). If those three Reviewers just come along at the right time, it makes a nonsense of your scheme. More "reviews" will be available for robo-reviewers. – Bill Woodger Jan 23 '14 at 11:34
  • 1
    @BillWoodger: As I said, it won't fix everything. Enough bad reviewers can still mess up any review scheme. I think it would still be an improvement over what we have now. – Ilmari Karonen Jan 23 '14 at 11:35
  • 4
    The only thing that will fix robo-reviewers is a cattle prod. – slugster Jan 23 '14 at 11:43
  • 1
    In the fewer than 20 seconds it takes to read the edit, note that "i" and "thanks" have not been changed and balance that against the "better formatting", three people have already Approved. The same will happen with your scheme. It just gives more grass for the robo-reviewers to chomp on. Any minor improvement will be outweighed by encouraging more robo-reviewing, which will in turn reduce any minor improvement. – Bill Woodger Jan 23 '14 at 12:12
  • This would probably be okay on a behemoth site like SO, but on the small sites with (many) fewer active reviewers, I'd be concerned that edits would be stuck in limbo for significantly longer. – ale Jan 23 '14 at 13:08
  • @AlE: Maybe, but I doubt it would matter much; see the expected review count calculation above. It could be an issue on really small beta sites that don't have more than three or four reviewers, but those sites generally rely heavily on diamond mods for review anyway. I don't think it would be an issue on any graduated site, or even on bigger beta sites like, say, Cryptography or Code Golf (which I have some experience of reviewing on). – Ilmari Karonen Jan 23 '14 at 13:34
16
+50

I pretty much go through the review queue skipping everything but the really good edits and the ones that need rejecting. I rarely hit the review limit for this reason. When I reject an edit that is awful, I watch it to make sure it gets rejected, and if it doesn't I go fix the post myself.

I'm very pro any idea that gives careful reviewers an advantage over the careless, but I'm not sure this suggestion would help to correctly reject bad edits. I'm not going to talk about rejections because the number of incorrect (IMO) rejections I've come across is negligible when compared to the number of incorrect approvals.

In the case of bad edits, if the majority of reviewers are good at what they do, then this suggestion helps, because with each additional reviewer that looks at an edit, the correct decision to reject becomes more and more likely to be made.

If on the other hand the majority of reviewers are more than a little approve-happy, then with each additional review, the more and more likely it is that it will be approved.

I have absolutely no proof to back it up, but I firmly believe that more reviewers are careless than are not.


Your edit makes things a bit more interesting, but the real trouble with this suggestion (and any other "improvement" that I can think of) is that it's combatting the impact bad reviews have without trying to improve the quality of reviews.

IMO we shouldn't be trying to think of ways to disadvantage careless reviewers, we need to think of some way to reduce the number of careless reviews happening in the first place.

That is a much more difficult problem to solve, and I certainly don't know how to solve it.

  • I... sincerely hope this is not actually the case. If it is, then we're basically SOL no matter what we do. Still, +1 for the thoughtful response. – Ilmari Karonen Jan 23 '14 at 11:57
  • Hopefully I've just become cynical :-( – OGHaza Jan 23 '14 at 12:02
  • No, that is fully my experience (and worse) and I do the same as you, go back and fix. It is a long, long time since I've hit the daily review limit. Not since I realised how much time I was wasting being cut apart by the bad reviewers. I intend to stop at 995 reviews. Don't need no stinkin' gold badge if the search for the thing, by those who just want gold, causes me so much gip – Bill Woodger Jan 23 '14 at 12:18
  • @BillWoodger: Perhaps you'd like to vote up this old feature request, then? (FWIW, I just did.) – Ilmari Karonen Jan 23 '14 at 12:39
  • @IlmariKaronen, thanks, have done. I'll go through the answers later as well. – Bill Woodger Jan 23 '14 at 12:43
  • 1
    @OGHaza: Regarding your own edit, I agree that it's a difficult problem. We've tried audits and they help, but not enough. Removing the gold/silver review badges, or at least changing them so that one or two failed audits would reset all progress, might do it -- it's hard to tell without trying it. All that said, however, I wouldn't be quite so pessimistic: the review system we have, flawed as it is, is still doing a much better job than having no edit review system would. And the proposed change should improve it, by making it easier for good reviewers to counter bad ones. – Ilmari Karonen Jan 23 '14 at 16:24
  • ...in general, I don't think there's going to be any single silver bullet that would magically "fix" the review system once and for all. To make it work better, we're going to need all those things together: we need better audits, we need to get rid of inappropriate incentives, we need to make the approval threshold more robust, we need easier reviewer review, we need better automatic bad edit detection... all of that, and more, and it probably still won't be perfect. Some crap's still going to get approved, and sometimes good edits will be rejected. But it'll be better than nothing. – Ilmari Karonen Jan 23 '14 at 16:29
  • Hmmph, so much for bounties attracting more attention. :-/ Here, have a free +50 rep. :-) (Just to be clear, I'm not trying to be snarky here. I do think this is a good answer, and easily deserves the bounty.) – Ilmari Karonen May 5 '14 at 11:36
  • @IlmariKaronen Haha, well it scored me a couple of up votes ;) thanks. I've long since abandoned the queue. – OGHaza May 5 '14 at 11:59
  • 1
    @BillWoodger, I've been sat at 997 suggested edit reviews for a couple of months now after reading your comment. – OGHaza May 5 '14 at 11:59
  • 1
    @OGHaza Me at 995. I think when the processing of the Close Vote Queue stabilises, there should be a push for some work on the Edit Review. Until something is done to improve things, I just don't want to do anything on that queue. Idiotic edits get routinely approved. I was editing from outside the queue, using the queue as a source, but just so much tosh. My pin-prick improvements are a good use of my time? So stopped that as well. – Bill Woodger May 5 '14 at 12:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .