160

A worryingly high number of bad suggested edits get accepted. For example:

(These are just examples of a few common patterns. The opposite phenomenon exists too.)

What can we do to educate reviewers to do a better job?

This is an open question, not a feature request, because I don't have a good answer. Some possibilities:

  • Provide a way to notify the reviewers (“hey, check that proposed tag wikis aren't plagiarised”).
  • Make the workflow for approval as long as the workflow for rejection, so that people don't blindly approve just because it's less work. (I don't know if some reviewers tend to do that.)
28
  • 5
  • 6
    A simple but probably not that helpful idea would be to increase the number of people that have to review an edit...but that just makes clearing the edit queue even more problematic and doesn't guarantee reviewers pay any more attention than they currently do.
    – Ben Brocka
    Jun 26, 2012 at 21:22
  • 1
    @BenBrocka We don't have enough eyes on the queue. The per-day per-person quota has been increased (to 40, and now to 50), which doesn't improve the quality of reviews even by well-meaning reviewers. Besides, more eyes would only improve reliability if the proportion of bad reviewers is sufficiently small: is it? Jun 26, 2012 at 21:24
  • 5
    @DaveNewton For each pattern I took one example out of many. #3 is not clearly homework, and the homework tag should not be summarily added to other people's posts. #4 is illegal (Wikipedia content requires attribution), and not helpful besides. Jun 26, 2012 at 21:26
  • 3
    @DaveNewton The discussion is still open, but most people seem to be in favour of getting rid of the homework tag. It's absolutely pointless, and probably actively hurting the overall quality of the site, as it's mostly used as an excuse for crap questions.
    – yannis
    Jun 26, 2012 at 21:28
  • 2
    The block in your second example is an error message, not code. The edit was a slight improvement over the original post, just not as good as a later edit. Jun 26, 2012 at 21:30
  • 1
    While I'm sure bad edits are a problem, I seriously doubt that the majority of edit votes are incorrect, which would be the only situation in which more reviews make it less likely that approvals are correct. And if the majority of edit votes are incorrect then the whole system needs a massive overhaul (but I seriously doubt that's the case)
    – Ben Brocka
    Jun 26, 2012 at 21:33
  • 42
    For one thing, we should stop allowing anonymous suggested edits. I can't recall more than a few that were even attempts to follow SE's guidelines. Random internet users who don't participate here do not know how we work. We shouldn't even give their edits a chance at being approved.
    – user154510
    Jun 26, 2012 at 21:34
  • 12
    @BilltheLizard The block was an error trace. The edit destroyed the line breaks in the source, making the trace unreadable and difficult to edit back into shape. This particular suggestor was fond of such bad edits, I remember rejecting many of his edits and going back to fix some of the posts he'd crapped on. Finally, one day, he got the Copy Editor badge and stopped editing… Jun 26, 2012 at 21:37
  • 6
    After we fix this lets fix wrong answers from getting upvotes or accepted Jun 26, 2012 at 23:44
  • 9
    You know, I have to call this one out: stackoverflow.com/suggested-edits/344474. That's a moderator approving 100% plagiarized text. Would it be possible for the tag wiki process to quickly search Google for similar phrases, and reject it automatically when there is a very close match? I've all but given up on reviewing edits when the reject votes just get overridden. Aug 8, 2012 at 12:50
  • 1
    @LittleBobbyTables That's a good idea, and probably worth posting as a separate feature request. I usually search if I see something that looks like ad copy, but I'll bet I've approved similar tag wiki edits in the past. Aug 8, 2012 at 12:53
  • 3
    @LittleBobbyTables: Its their ad copy. You know, the stuff they want to get out there on the internets, that pimps their products. Not exactly the kind of stuff you jealously protect with IPs. I took a minute to add an attribution. Just in case their ad folks are upset that their copy is being given prominent placement on a much frequented website.
    – user1228
    Aug 8, 2012 at 14:52
  • 21
    Is the summer of love over yet? I'm looking forward to the Fall of our discontent.
    – user1228
    Aug 8, 2012 at 14:53
  • 9
    @Won't Shhh, don't let the kids know this Summer of Love business is only a pretence to lure more of them in and use them as cannon fodder in the Fall of Sarcasm.
    – yannis
    Aug 10, 2012 at 10:04

14 Answers 14

82

One of the problems with the suggested edit system is that it takes almost zero time or effort to submit accept votes, or for that matter, reject votes. It takes much longer to do the right thing and improve bad suggestions.

I often try to improve suggested edits. Sometimes, they're good suggestions where the editor fixed 50 typos and just happened to miss one, but that's rare. In the vast majority of cases where I click Improve, the original suggestions are unhelpful, weird, severely incomplete or even actively harmful, and I uncheck "suggestion was helpful."

Unchecking that box is so futile that I'm not sure why I bother. Every time I improve a bad suggestion, it gets accepted while I work. It's gotten to the point where I'm discouraged from using the improve option at all.

Possible solutions (could be combined):

  1. Don't act on accept/reject votes as soon as they're cast (or as soon as two agreeing votes are cast, in the case of SO). Force the suggestion to sit in the queue for n minutes.
    • If an improvement is submitted at any time, accept it immediately (like the way it works currently).
    • After the timer is up, if the total number of votes — both accept and reject — has reached some threshold, go with the majority.
    • Otherwise, add m more minutes to the timer and hope for more people to review the post or for someone to improve it.
  2. If and when someone suggests improve, stop accepting accept/reject votes on that suggestion. Don't even show the suggestion in the queue anymore.
    • If the improvement is submitted, accept it immediately (like the way it works currently).
    • If the improvement is canceled, start showing it in the suggestion queue again.
    • After p minutes, if the improvement hasn't been submitted, assume the improver abandoned the attempt to fix the suggestion and start showing it in the suggestion queue again.

Another problem with the current system is the lack of feedback for participants. Suggestion reviewers have no way to know whether the community agrees with their decisions, so bad reviewers keep on obliviously doing a bad job. As long as reviewers are approving bad edits, users who suggest bad edits won't know that they need to get better, let alone want to improve, and none of the rest of this post will matter.

Suggestion submitters have it better, but only because the bar is so low. They can find out whether their edits have been accepted by revisiting the posts they edited, but only after the edits are reviewed, and who knows how long that takes? They can also check their rep, but that only tells them about approvals, not rejections, and only for up to 1000 rep, and even then only while they're under 2000 total rep. Those who repeatedly suggest bad edits can get auto-banned, but the ban isn't designed in a way that fosters improvement.

Possible solutions (could be combined):

  1. Give editors more feedback about whether their suggestions are good. In the case of rejected edits, be sure to explain why, perhaps with blue-level notifications.
  2. Allow the community to mark low quality suggestion approvals, and give feedback to the editor and reviewer(s). (This has been suggested before, even at this very question.)
  3. Make personal review stats more prominent in user profiles. This could be a hidden section, like the current helpful flags and votes sections.
  4. Add an intermediate step between "suggest all the edits you want" and "you're banned from suggesting edits for a week" that somehow tells the user "a lot of your suggestions are getting rejected for reasons X, Y and Z; you might want to reread the edit guidelines."
  5. Change the reviewer requirements from being based entirely on the edit privilege to being based partially on whether the reviewer's previous accept/reject votes were correct. (Partially inspired by this idea.) This one went through numerous iterations in my head and is probably too complicated to be implemented, but maybe it'll help with brainstorming. Actually, tracking reviewer quality may not be as far-fetched as I thought! Some kind of "audit" system is in the works, per dev Kevin Montrose.
13
  • I don't think that your first solution makes any sense, n would vary a lot depending on the time of week and random factors. Your second proposal does make sense, and might work. Aug 12, 2012 at 21:35
  • 13
    Sometimes I first reject, to give feedback to the submitter (not that it does much good since the submitter can't see that feedback without going through hoops), then click Improve and make sure to uncheck the “helpful” box. But these days, merely typing a custom rejection reason can be enough time to let a crappy edit be accepted! Aug 12, 2012 at 21:38
  • @Gilles I think perhaps I was not clear in explaining my first proposals, and I have edited. Could you check again and let me know if you still disagree?
    – Pops
    Aug 12, 2012 at 21:42
  • I don't see how waiting for another user to intervene for a fixed duration can be a good thing here. There's no way to know how long it will take, and it is better to have edits reviewed as quickly as possible. By the way, I see feedback to editors as indispensable; sadly Jeff is against. Aug 12, 2012 at 21:48
  • 3
    Okay, I can understand that point of view. I just think the current problem, where bad edits are reviewed quickly and accepted quickly, is worse than the artificial time limit.
    – Pops
    Aug 12, 2012 at 21:51
  • 1
    I think #2 is key. Improving a post should always take priority over approving as-is. Help users discover and learn from the results of their edit suggestions looks relevant to your interests as well.
    – Brad Mace
    Aug 12, 2012 at 22:34
  • 1
    @Gilles Getting suggestions approved as quickly as possible is exactly the problem, as Popular Demand points out. People are reviewing them as quickly as possible, by clicking Accept on almost everything that comes up. Getting it right is what we need to worry about.
    – Brad Mace
    Aug 12, 2012 at 22:38
  • @BradMace Having suggestions approved is the problem. But having them (correctly) reviewed quickly is a goal of the system. Aug 12, 2012 at 22:40
  • 9
    To /review/review!
    – Tim Stone
    Aug 13, 2012 at 19:51
  • 1
    @TimStone heh, yeah, I did think about how silly "meta-review" seemed while I was writing. But we wouldn't actually need a /review/review if we had, for example, the timer system. I have been thinking for a while that the privileges system should be based more on merit than reputation, but I realize that major changes in that direction aren't going to happen any time soon, if ever.
    – Pops
    Aug 13, 2012 at 20:04
  • 1
    While I made the comment in jest, I actually don't consider it that silly at all. I'm not sure why people are showing up to review things if they're just going to rubber stamp them, since that's arguably even more harmful than the queue filling up. Having a way to educate or otherwise stop those people seems like it'd be beneficial, problems with the original editors aside.
    – Tim Stone
    Aug 13, 2012 at 20:13
  • 2
    Well, if it were all silliness and no substance, I wouldn't have suggested it. But there is one legitimate concern: how do we know that the /review/reviewers are going to be any better than the /reviewers? Do we need /review/review/reviewers? Before you know it, it's turtles all the way down. Meta-review in any form is, at best, a bandage. What really needs to happen is exactly what you (and I) have said: people learning how to review properly.
    – Pops
    Aug 13, 2012 at 20:25
  • 1
    Potentially related to reviewer-review: Review-beta: Obviously good answer in low-quality
    – Tim Stone
    Aug 13, 2012 at 20:51
48
+50

The only thing I can think of would be to allow users to flag a specific revision when they discover a bad edit that has been approved. If enough of these flags against a specific reviewer are found valid, give them a time-out from reviewing.

I think the timeout should probably be longer for bad reviewers since it's harder to spot when it happens unless you were also reviewing the same edit at the same time. Or maybe just a lower threshold before getting banned from reviewing.

Also, the reviewer should get some sort of notification when their edit is deemed bad so hopefully they can improve before a timeout becomes necessary.

1
  • 23
    I'd be all for a flag revision as incorrect, or possibly even better, flag acceptor as careless. It's bad form to call people out, but I can easily name a half-dozen users who habitually accept bad edits. Aug 8, 2012 at 0:48
32
+200

A very simple thing nobody seems to have mentioned so far: provide basic instructions for the reviewers. The current instructions are quite minimal (after clicking "more"):

  • Approve edits you know are correct
  • Reject those you know are wrong
  • Improve to improve this suggested edit
  • Not Sure if you are unsure and want to skip this suggested edit

What is considered correct or wrong? It's not clear. Those words could at least be linked to a FAQ, or Meta discussion. Not to mention the description of "improve", which basically says "improve to improve"!

I don't have any suggestions for new instructions right now, but I'm sure we can come up with something better (and concise), given enough discussion.

Also, I don't believe this will solve the problem completely, but it's a step forward anyway.

UPDATE: I've created a Review Guidelines faq proposal. It's still a draft, and any contribution from the community is welcome.

5
  • 5
    There should probably just be canonical, "how do I review posts?" and "what kinds of edits should be approved/rejected?" questions on meta that are linked from the review system, and shoved right in the faces of people when they first start using the system. Then they can just be edited over time by the community.
    – Servy
    Aug 15, 2012 at 17:43
  • 3
    How can we present these expanded instructions in a way that will actually lead to people reading them? Experience has shown that the users who need the most guidance are the least accepting of it.
    – Pops
    Aug 15, 2012 at 18:01
  • @PopularDemand Maybe the instructions box should default to the expanded state, at least for users who reviewed less than x questions. It does not guarantee they'll read it, but better shoving it in their faces than hiding it under a "more" link. And, I'm totally for your suggestions of giving editors more feedback, and allowing the community to mark low quality edits. Those should make at least some people seek more information on how to edit.
    – bfavaretto
    Aug 15, 2012 at 23:01
  • I sooo agree with this. I even tried to do something similar back in the dark ages. And I have to admit that if I've largely turned my back on the different review queues, it's because I too often just don't know exactly what I'm supposed to do. (Or editing crap into shape is too much hard work...)
    – Benjol
    Nov 27, 2012 at 10:41
  • Correct minor edits really need to be given clarification in these minimal instructions. Otherwise editors who have read what's on meta will know to favour rejection or improve and yet we continue to have lots of suggested edits getting half a dozen various accepts, rejects and edits with a random one winning, and someone editing, as well as everyone who goes back to review the outcome, being frustrated with the final result.
    – Mark Hurd
    Jul 11, 2014 at 6:50
24

Stop tying the edit/review privilege to reputation. I'm convinced there is no correlation between the amount of reputation you have and your proficiency at editing.

You can learn to be a good editor, but only by editing a lot of questions and getting feedback on them, not by earning reputation. Someone can get the necessary reputation to review and edit without ever editing a single question.

Instead, I think the edit/review privilege should be tied to a number of approved suggested edits with a decent approved/rejected ratio. For example, say a minimum of 250 edits with a 90% approval rate.

Approving invalid edits does way more damage than just letting a few bad questions go by; it teaches new users the wrong way to edit and review.

5
  • 3
    IIRC a developer said it was difficult to tie privileges to something other than reputation, I can't find the post now though. A minimum of 250 edits is unworkable outside SO, I don't think there can be asingle number across SE, which makes this even more complicated. Aug 8, 2012 at 1:21
  • @Gilles The reputation to acquire a privilege can also differ across sites. E.g. "create tags" requires 1500 rep on SO but only 300 rep on IT Security. So from an end-user's perspective it shouldn't be a problem. Aug 8, 2012 at 6:15
  • 1
    Related: this question and this question. Aug 8, 2012 at 6:22
  • @Gilles I disagree that 250 edits is too hard outside of SO. I have around that many on Mi Yodeya -- more if you count tag-only edits, and I've been around less than a year.
    – Scimonster
    Jan 1, 2015 at 17:32
  • @Scimonster It's not a matter of whether it's doable, but whether enough people can reach the target so that there are enough people manning the review queues. That's especially true on a young site which doesn't have a lot of old material needing to be brought up to date — for example Emacs (~3 months old) has a single user with 250 edits but 17 with 2000 rep, and 35 with 1000 rep (the threshold on beta sites). Jan 1, 2015 at 17:46
19

It seems like most robo-reviewers accept because that's just one click, as opposed to three for rejecting. Therefore, making it take more clicks to approve a bad edit should slow down robo-reviewers.

If an edit has any reject votes, show a warning while clicking approve. It would say something like "This edit already has x reject votes. Are you sure you want to approve it?" It would have an "approve anyways" button there. Then there could be a line, and the standard reject form right there.

I'm not going to draw up an image, so here's a plaintext rendition of what it could look like:

This edit already has x reject votes. Are you sure you want to approve it?

<approve anyways>


Reject this edit

  • spam or vandalism This edit defaces the post in order to promote a product or service, or is deliberately destructive.
  • no improvement whatsoever This edit fails to make the post even a little bit easier to read, easier to find, more accurate or more accessible. Changes are either completely superfluous or actively harm readability.
  • irrelevant tags This edit introduces tags that do not help to define the topic of the question. Tags should help to describe what the question is about, not just what it contains.
  • clearly conflicts with author's intent This edit deviates from the original intent of the post. Even edits that must make drastic changes should strive to preserve the goals of the post's owner.
  • attempt to reply [1] This edit was intended to address the author of the post and makes no sense as an edit. It should have been written as a comment or an answer.
  • causes harm []

<reject>

This could also apply to other review queues.

16

One possible solution is to make it a bit harder to approve and a bit easier to reject to edits that come from a user with very high rejection ratio :

if(reviewer is OP){
    1 app./rej. gets approved/rejected //current behavior
}else{
    if(Editor has at least 10 edits){
        if(  1/10  to   0/10 rejection) 2 app. gets approved, 3 rej. gets rejected
        if(2.5/10  to   1/10 rejection) 2 app. gets approved, 2 rej. gets rejected //current behavior
        if(  4/10  to 2.5/10 rejection) 3 app. gets approved, 2 rej. gets rejected
        if(  10/10 to   4/10 rejection) 3 app. gets approved, 1 rej. gets rejected
    }else{
        2 app. gets approved, 2 rej. gets rejected //current behavior
    }
}

Numbers might change, but I think they reflect the most common types: perfect, average, needs some work and bad with their appropriated protection/handicap

1
  • something like this has been suggested in bypassing the review queue for avid editors: "edit weight number / feature that would arrange suggested edits queue so that suggestions from "established editors" tend to flow to top... It takes only a minor effort to rephrase current description of flag weight to make it"
    – gnat
    Aug 18, 2012 at 7:29
11

One rather simple step would be: Do not allow to suggest a rejected edit again. I have seen that: Edits I and others had rejected were just resubmitted by the same user until they found a bad reviewer.

Solution: If an edit was rejected do not allow the suggester to edit that post/wiki again for 24 hours. That should be rather easy to implement I guess.

9
  • 5
    That makes sense, but it's a rare case. Most bad editors edit a post and move on, and don't even know they've had edits rejected. Aug 8, 2012 at 1:11
  • 1
    @Gilles Your source? I don’t think this is monitored (enough).
    – fuxia
    Aug 8, 2012 at 1:12
  • 1
    There is no notification for rejected edits, and it's tedious to find out even if you want to know. A notification has been requested. I've reviewed close to 10k edits across SE, and I've never seen that focus on one post, except on tag wikis, so I'm sure it's a rare case. Aug 8, 2012 at 1:16
  • 1
    @Gilles There's no notification? How are we expecting people who make bad edits to learn anything if they're not notified of the outcome?
    – Brad Mace
    Aug 8, 2012 at 18:18
  • 2
  • @toscho well consistent rejection has certain effect: "repeated edit suggestion rejections from multiple users will cause your edit suggestion rights to be suspended for (n) days, where n is currently 7" (quote source)
    – gnat
    Aug 18, 2012 at 18:58
  • @gnat I was talking about not consistent rejections.
    – fuxia
    Aug 18, 2012 at 19:03
  • 2
    @toscho it's not hard really. Six rejections can trigger ban - this has been my own experience about a year ago
    – gnat
    Aug 18, 2012 at 19:34
  • 1
    In my case, I try to learn from every of my edits to ensure I do not repeat mistakes. However, I get very frustrated when I get unfairly rejected when I'm 95% sure I was right. I must admit, I did submit twice some Edits in the past because of that. You got a good point, but that might not be solving the real problem IMO.
    – ForceMagic
    Dec 18, 2013 at 8:05
11

I was going to post this as a separate question until I came across this topic. If something seems off about the wording it's because it was meant as a standalone post.

I believe my point of view differs from most of the already provided answers in that it focuses more on raising awareness rather than reprisal on the offenders.

The issue I'm having with the suggested edit review queue is very simple: people are encouraging insufficient edits by approving them anyway.

I have seen a lot of edits passing by that either only partially fix something or plain deface the post that end up getting approved. The following are a few samples of edits in the last hour (2 different users), but this certainly wasn't a temporal surge of low quality edits.

Situation 1

https://stackoverflow.com/review/suggested-edits/3667680

At first glance this doesn't seem too bad. Then when you look more closely:

  • Language tag in title
  • Incomplete code formatting (Student and Course remain unticked)
  • None of the English has been fixed

When you look at the total revisions for this answer you can see it needed 3 more revisions to get in a decent shape.

Situation 2

https://stackoverflow.com/review/suggested-edits/3667621

  • Random code formatting
  • No English fixed
  • "Thanks" not removed

Once again a case of just clicking the {} button a few times and leaving everything else in there

Situation 3

https://stackoverflow.com/review/suggested-edits/3667617

  • Language tag in title
  • Random code formatting
  • Error message in code format, making it hard to read

The only substantial edit isn't a particularly useful one.

Situation 4

https://stackoverflow.com/review/suggested-edits/3667520

This one actually got rejected but it still got 2 approval votes.
It literally has nothing going for it:

  • Horrible title unfixed
  • Random code formatting
  • Added incorrect English
  • Removed 2 lines of whitespace, left 3 others in there and didn't touch indentation

Still he got 2 approval votes.

Situations 5, 6 & 7

https://stackoverflow.com/review/suggested-edits/3667714
https://stackoverflow.com/review/suggested-edits/3667686
https://stackoverflow.com/review/suggested-edits/3667650

Each of these edits got approved while the only thing they do is put jsfiddle links behind a word and bolden a few words that were already distinguished by separation.

In fact there are still several (minor) issues that aren't touched at all. If these things were valid enough to let trough then the other stuff should be valid enough to reject the edit as well:

  • Code formatted with bold instead of code formatting
  • Irrelevant tag
  • Code not indented

What am I trying to reach with this post?

I believe there are a few issues here.

  • I waste time by both correcting the original post and the suggested edit.
  • If it gets approved before I get my hands on it, people will read a post that has poor quality.
  • A question gets pushed to the top despite nothing substantially being changed.
  • Insufficient edits are indirectly encouraged because seemingly many of them are being approved.
  • The trustworthy representation of a user is inflated because he gains reputation without adding value to the network.

I believe the main offenders here are the people who approve these suggestions and subsequently encourage this behaviour. We as reviewers have the possibility to stop these edits from entering the system but seemingly we're failing as a group at this task a substantial amount of time.

I won't deny that I might have set the bar higher than is commonly accepted, but I strongly doubt that these suggestions should've been allowed. Edits should ideally address all issues in a post whereas the samples I posted definitely don't do that. Either they plain deface the post and perform one specific and often incorrect action (person 1) or they invent problems and make unimportant changes.

What can we do?

I think it's important to provide feedback to people about how they voted. As far as I'm aware we don't have any possibility to see how the rest of the reviewers voted on an edit unless you explicitly keep track of the edit. This is a lot of work that will require a lot of tabs to keep open and shouldn't have to be done in the first place.

Those that suggested an invalid edit should receive a notification when their suggestion got declined. I haven't done any suggested edits in my pre-2k period so I don't if this is possible (let me know if it is), but if the "custom message" option actually gets shown to the person himself, we can use that as a way of communicating about what he should change in the future.
This would be mainly aimed at the people who go on invalid suggestion spree, usually you get a few of them if you're on a review spree.

Then again, messages might not be enough. As it is now there is a formula in place that blocks people from suggesting edits for a 7 day period if they get too many rejects.

The formula is

(rejects - (approvals / 3)) >= 5

This is in itself a good formula but I believe we can do more with it. When it comes to adjusting behaviour, going from no notion to full stop is rarely a good idea. People will get banned for too many rejections and they're basically left with one feeling: why wasn't I told in time?

I think that's exactly what should happen. As soon as you reach certain tresholds you should get a warning that you're getting too many rejections. By working in stages and positively approving people when they become more in-line with the rest of the community it's a lot easier to correct this (we'll bypass any philosophical discussions by assuming the community as a whole has the correct judgement).

By adding these warnings users will more quickly be notified when they're getting rejected a lot and will be able to improve their suggestions sooner. Likewise they will have the feedback of the positive notifications from the system when it tells them their suggestions are getting approved a lot more.

In short:

  • Give us a result of our votes. Either instantly or at the end of the day let us know how the cases we ruled on were decided by the community.
  • Let users know when their suggestions were rejected and why.
  • Provide warnings when a user is getting close to the lockout period.
  • Provide a positive notification when a user's aproval rating is rising.
4
9
+100

For the record, the % of rejected, approved, and contested edits has stayed fairly static over time - but the totals have increased considerably as folks have become more comfortable with these systems. So, as with everything else, the tooling must improve along with them.

What we're doing about this:

  • Short-term: increasing the number of reviews required. In practice, it seems that approving these was just too easy - more eyes on them should help.

  • Medium-term: Implementing this. Better if fewer bad edits are submitted, period.

  • Long-term: continue to monitor this, and listen to suggestions from y'all.

7
  • 4
    Do you think it's going to have any positive effect? I don't see a shortage of bad reviewers. Also, why did you increase the number outside SO? My impression outside SO (or maybe outside the trilogy) was that reviewers tend to be more dedicated users, and the only really bad decisions are the Wikipedia dumps in tag wikis, which even most ♦ approve. Oct 16, 2012 at 20:23
  • @Gilles: I expect the effect to be fairly small, whether good or bad. The number of contested approvals on SO is about 15%, up from 7% at the start of the year (contested rejections are up as well, but by significantly less). The reason for enabling it outside of SO is that without multiple reviews it's pretty hard to track contested reviews. Which means I really can't tell you how many contested approvals or rejections occur outside of SO because it's not normally possible to contest them! If this is a problem, I want to know about it - anecdotes don't go very far.
    – Shog9
    Oct 16, 2012 at 20:40
  • Ok, that makes sense. I still wonder how you know that uncontested reviews are good — I know I've seen plenty of bad ones, but I can't give a reliable statistical estimate. Oct 16, 2012 at 20:47
  • Short of looking over every single one and recording my own (always objective) opinion, there's no good way to judge "goodness" short of looking for some action that indicates disagreement (opposing votes, rollbacks, etc.). Opposing votes are the best thing I have right now.
    – Shog9
    Oct 16, 2012 at 20:49
  • 2
    The right thing to do would be to go over a random sample and classify them into clearly wrong/borderline/clearly right. I've thought of doing that myself but 1. I never found the time and 2. why would you trust me on this anyway (I've done a lot of review, but by my own argument that doesn't indicate that I'm good at it). Oct 16, 2012 at 21:07
  • @Shog9, do you want me to start a hall of shame? In the last hour I've seen at least 3 approvals of rubbish edits which were 'uncontested' simply because eejit reviewers got there before I did...
    – Benjol
    Feb 13, 2013 at 7:39
  • For now, roll these back @Benjol - and if you see the same reviewers routinely messing up, flag.
    – Shog9
    Feb 15, 2013 at 2:47
4

The low number of suggested edits being left in the queue after the launch of the beta review system is obviously not as good a sign as we may have thought, and one of the contributing factors to both that problem and the one presented here is the lack of effort required on the reviewer's part. But at the same time, the beta review system is contributing to the lack of effort. It's really a hilarious catch-22.

But the problem doesn't lie only in the fact that no effort is needed to approve an edit, but also in the fact that the users making the suggestions don't really know what they're doing.

I just saw this suggested edit, and I don't know who to blame. Do I blame the reviewers for doing an awful job, or the suggester for making an edit that detracts from the post (the URL references threw me off in the rendered view ... only the markdown view gave me an understanding of the strange situation).

So it becomes an issue of who to educate better. In the above case, I lean more towards the suggester — with better knowledge of markdown, that edit could have been pretty helpful (to me, at least; linking to documentation is always good). The intent of the edit was helpful, and from first glance, I can see why the reviewers approved it (they don't have a particularly bad approval history).

But then you have cases like this suggested edit which replaced a link to one of a YouTube video ... and got approved. In this case, I obviously lean towards the reviewers. God knows what crossed through their heads, and it was more than likely an effort towards keeping the queue clear (though I don't see why that would be beneficial to them, since there is no reward for approval).

So we're at an impasse — I don't believe there is one consistent way in which we could eliminate the problem in terms of educating users better. Plus, I think whatever systems we have in place for helping users learn the system are helpful enough.

But I'd like to bring up the argument about whether there is a correlation between high reputation and good reviewing skills. A user here argues that the correlation doesn't exist, that a user can get enough rep to approve/reject without having edited once.

This is true, sure, but chew on this: the only other way to earn reputation is by asking good questions and posting good answers (and suggesting tag wiki edits if below 5000, but +2 is inconsequential). Both of those are indicators — to me, at least — that someone knows what makes up a good question/answer, and that I could trust them approving good edits and rejecting bad ones.

But 2000 rep is definitely not a high enough threshold to determine whether a user really understands the difference between a good and bad question/answer. So my proposal would be to raise the reputation needed to review suggested edits (and, in turn, since the two privileges are associated, the reputation needed to edit freely) to 5000. At this threshold, we can at least minimize the number of bad reviewers (because 5000+ tells me that they have a pretty good question/answer quality history) and the number of bad edits approved.

(And in case anyone brings it up, I'm pretty sure this user, who approved the spam YouTube edit, is either compromised or did it purely for kicks...the world may never know).

3

People make mistakes. You can't avoid that. The suggested edit interface is easy to use. But unfortunately, it is easier to accept than to reject (you have to give a reject reason). And when in doubt people tend to the positive side (at least the helpful ones).

Besides, if a post already has a accept or decline vote, it is psychologically attractive to give the second vote: clears the queue with one.

But as I look at the size of the edit queue, I don't think there are much people active on this. So maybe some education can be helpful. Maybe a small e-course for reviewers.

1
  • 11
    Redacted had 55 edit suggestions approved, and 39 edit suggestions rejected. I'd say some users fail about as often as they succeed...Yet their edits keep getting approved "by mistake".
    – user7116
    Jun 26, 2012 at 21:51
2

In addition to anything else, or as a separate measure, I think that one way to prevent bad edits from being accepted would be to allow the person who made the original post to approve or reject the edit before anyone else has the chance. If they don't do anything after a certain length of time has passed, or if they actively assert that they don't want to go either way, put the edit into a review queue to go through whatever other process is in place. I think that this would work because the original poster is more likely to understand and care whether the suggested edit is an improvement, also it gives them a more direct chance to learn about how questions or answers should be articulated and formatted.

Just to clarify, here is the workflow I have in mind:

  1. User A posts a question or an answer.
  2. User B decides to edit the post made by user A
  3. User C comes along and decides to edit the post, but cannot because User A has not done anything yet
  4. User A has the opportunity to Accept, Reject or Skip the edit
    • If User A accepts the edit, it is applied to the post
    • If User A rejects the edit, the changes are rolled back and the post can be edited by User C
    • If User A chooses to skip the edit, the post goes into a review queue
    • If User A does nothing after a certain length of time, the post goes into a review queue
1

I actually signed into meta to answer this. Here is what can be done:

After an edit is peer viewed and approved, let the edit be notified to the original user and ask if he would accept this as an edit and the reason for edit.

Ofc. User whose post is edited must have some reputation to get this privilege.

Every edit which is not accepted by the original poster has to result in edit going back for peer view (hopefully, this time someone else) and depending on that result, reputation has to be negated from the first editor.

Confused?

Again...

ABC posts a question, PQR edits it, if ABC has reputation of 100 (say for example) he can see that his post is edited with the reason for edit and ask if he likes the suggested edit, if he likes it, edit was good! and has been approved by an upcoming stack member, if it is rejected or disliked by ABC only then, this time, edit goes to XYZ. If XYZ approved that PQR's edit was good, PQR gets +rep else -rep.

What will be achieved by this?

  1. Member who edit will have feedback on HOW NOT TO edit rather HOW TO edit.
  2. Member who got his post edited and perhaps liked the edit will know how to post it in future. At the moment, when I was new (see my stackoverflow profile, I have joined recently) Someone edited my post and I kept wondering as to what was changed!!! Trust me, I still do not know! However, learnt it across on how to post keeping everyone in mind, even those who seek answers on search engines. If you could see I have edited few posts where members rep was way more than mine and also has longer history than mine, yet, the sense of posting the content has not been authored in their habits.

Answered, not for bounty but, to put things across. Good that you asked! :)

8
  • 5
    Well, generally, people who's posts end up being edited are often the people who don't know how to edit posts. Sure there are exceptions, but they aren't the common case.
    – Servy
    Aug 15, 2012 at 17:45
  • @Servy I wrote an answer based on my experience with stackoverflow.Sure there are trolls around everywhere including in meta, who downvoted my post without bothering to explain WHY Aug 15, 2012 at 17:48
  • StackOverflow would be the worst place to do this, IMO Aug 15, 2012 at 17:50
  • 1
    Hmmm... Perhaps, you had better stay there than me... However, giving a thought, there are few sensible people joining SO everyday especially the ones who are into android! Aug 15, 2012 at 17:51
  • 5
    Welcome to Meta! I think this has potential for edits involving code and/or wording changes, because the OP is really the only one who can judge against the post's original intent. On the other hand, it could be detrimental for spelling/grammar/formatting edits. If the OP doesn't visit the site often, review could take a long time. And since the OP made the mistakes in the first place, he's probably not highly qualified to judge the fixes.
    – Pops
    Aug 15, 2012 at 18:10
  • hmmm... interesting.... Aug 15, 2012 at 21:30
  • 1
    @KarmicDice Given that you're new here, please note that downvotes here are often used to indicate disagreement. So should you be confused about the downvotes you received, don't be. They most likely simply indicate "I don't agree with what you propose".
    – Bart
    Aug 15, 2012 at 21:36
  • 2
    Yea, I gathered that already! My Coding-ego is kept aside at meta :) Aug 15, 2012 at 21:40
-5

I've seen perfectly valid edits sitting with a single reject vote, so the inverse to your question is a part of the same issue -- What can we do to stop good edits from getting rejected?

If you can find examples of bad edits accepted, it stands to reason that there are good edits being rejected (in equal numbers?) as well, and are thus gone from our purview. Can one of the data wizards pull stats? What percentage of edits are accepted vs. rejected?

It seems to me that the more alarming of the issues you present are pertaining to questions or answers. Tags, being more visible to experienced users (because we use them on our own questions) will self-correct over time, IMO. A question could be silently polluted by a bad edit (or suffer for lack of a rejected good edit) and stay that way until someone "in the know" happens across it.

I have two suggestions, both involve using systems that are already in place to make sure that a set of qualified eyes are on an edit before a decision is made:

Suggested fix #1 (most restrictive) only show body edits for questions or answers for which the user has at least a bronze badge. This makes sense -- until I've received 100 upvotes on a given tag, what is the likelihood that I can accurately evaluate the quality of an edit to a question or answer on that subject?

We can continue to show tag wiki edits and question tag changes to anyone with review permission, but I think reviewing the -content- edits should be reserved to those who are demonstrably equipped to evaluate them.

Suggested fix #2 (less restrictive) 2 reject/accept votes required when at least one of those votes comes from a user with a bronze badge in the given tag, OR 4 reject/accept votes from all non-badged users.

Again, we can continue to show tag wiki edits and question tag changes to anyone with review permission, but we either require more "unqualified" votes or a "supported qualified" vote to change the body or questions or answers.

5
  • 6
    I'd asked earlier about rejected wrong edits; they're mostly rejected because some people reject bug fixes because they “change the meaning”. That's a different issue. I'm not fond of your suggestions; a lot of suggested edits are only about language and formatting so require no subject knowledge, and smaller tags have no bronze badge so editing in them would become difficult (even impossible with your fix #1). Jun 26, 2012 at 22:32
  • 3
    The downvotes have me fading out like Marty McFly, but I stand by my suggestions: you can't have your cake and eat it too. Either the bulk of edits are minor, in which case a few questionable approvals isn't a big deal, or we need (in your words) better-educated reviewers. The best way to have better educated reviewer is to use the site's existing mechanics to ensure they are demonstrably better educated :)
    – Chris
    Jun 27, 2012 at 1:42
  • 1
    But your answer doesn't say anything about having better-educated reviewers. Knowledge about a particular programming language isn't the same thing as knowledge about how to edit posts on Stack Exchange. Jun 27, 2012 at 1:47
  • 1
    The only other type of expertise available is in the English language, in which case we ought to place the entire burden of SE edit reviews on 10k+ users from Writers!
    – Chris
    Jun 27, 2012 at 2:10
  • I like your 3 first paragraph, however I don't like your suggestions because I think they are not valid for all cases. You got the other side of the coin problem though! :)
    – ForceMagic
    Dec 18, 2013 at 7:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .