10

There have been a number of highly voted discussions regarding incorrect suggested edits approved: first, second, third, fourth.

I think that often when bad suggested edit is approved, it actually gets 1 or 2 rejection reviews. And I think it's incorrect to mark edit as approved if it has 3 approval and 2 rejection reviews as it shows that actually there is no agreement between reviewers.

Proposal: If less than e.g. 80% of reviewers agree on approval/rejection, new reviewers should be appointed.

If an edit gets 3 approval and 2 rejection reviews, it means only 60% of reviewers agree so additional reviewer(s) should be chosen. And so on until number of reviews with the same decision is 80% of the whole number of reviews.

If e.g. 20 people reviewed the edit and there is no consensus between them (i.e. less than 80% people agree on it), it should be flagged for moderator attention as it means that ordinary SO users can't agree on it.

Improve this question

2 Answers 2

Reset to default
3

I agree with the general idea here, but I don't think it should be as high as 80%. 75% seems sufficient, and it could probably be a little bit lower. It should simply be strictly higher than two thirds (66.6...%). Then we avoid the current situation where a two-to-one disagreement is considered resolved.

That seems similar--after all, only a 5% difference. But it really is very different, because it means that if one reviewer wants one outcome, and three other reviewers want another outcome, the three other reviewers win.

This is important for two reasons:

  • It's bad for posts to stay in the review queue for many hours. What you've proposed would likely keep them there for days.
  • Editing a post takes it out of the queue, so there's not much point in requiring a large number of people to review a post.

Furthermore, I'm not sure it's even necessary to have a large majority. 51% percent might be sufficient, once several reviewers have weighed in. What I see as a problem with the current system is that random fluctations have too much bearing on whether or not an item is accepted.

I think if there are only 2 reviewers, and both have accepted or both have rejected, that is sufficient. I think that any situation that approximates that level of certainty is sufficient. 3 vs. 1 is definitely a sufficient majority.

Improve this answer
2
  • Agree. The problem is with the 3-2 approvals, as it only takes one click to approve but 3 to reject. In a photo-finish that's a real disadvantage!
    – Bo Persson
    Commented Mar 23, 2013 at 16:08
  • 1
    @BoPersson I see 4-2 as better than 2-1, and I think 4-2 approvals/rejections are probably fine. In a 4-2 approval/rejection, six people have expressed their views and the decision has been made by a significant margin.
    – Eliah Kagan
    Commented Mar 23, 2013 at 16:11
2

The most frequent case where I observed lack of consensus was on minor edits (e.g. an edit that only fixes a grammatical error, or add a source block, etc.). Other causes (e.g. invalid edits) do not frequently lead to conflicting reviews and are often voted unanimously.

From my experience both as an peer-reviews editor (when <2k) and later by comparing my revision votes with the final outcome, I found that some people would reject edits as too minor, while others would accept them (because they improve the quality of the site, even if minor). The line between each interpretation is subjective, and has been subject of an extended discussion:

My conclusion is that no change in the number of reviewers is required.

Improve this answer
1
  • 2
    You're right that for many disagreements over borderline too minor edits, the current system isn't really broken. Or, I should say, when it comes to those questions, the Stack Exchange design is not really broken. Usually, when an edit fixes just a couple small problems with a post but leaves others, the best course for a reviewer is to improve on the edit. (This is especially advisable for a reviewer who considers the post helpful and would otherwise accept it.)
    – Eliah Kagan
    Commented Mar 23, 2013 at 16:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .