What can we do to stop bad edits getting accepted?

Question

A worryingly high number of bad suggested edits get accepted. For example:

• edits that add bad formatting to random words
• edits that break the formatting of a block of code
• edits that add wrong tags
• edits that add Wikipedia dumps to tag wikis
• edits that clearly replace links with spam
• edits that only replace the actual URL with something unrelated
• edits that change the author's solution
• edits that add comments to posts

(These are just examples of a few common patterns. The opposite phenomenon exists too.)

What can we do to educate reviewers to do a better job?

This is an open question, not a feature request, because I don't have a good answer. Some possibilities:

• Provide a way to notify the reviewers (“hey, check that proposed tag wikis aren't plagiarised”).
• Make the workflow for approval as long as the workflow for rejection, so that people don't blindly approve just because it's less work. (I don't know if some reviewers tend to do that.)

Answer 1

One of the problems with the suggested edit system is that it takes almost zero time or effort to submit accept votes, or for that matter, reject votes. It takes much longer to do the right thing and improve bad suggestions.

I often try to improve suggested edits. Sometimes, they're good suggestions where the editor fixed 50 typos and just happened to miss one, but that's rare. In the vast majority of cases where I click Improve, the original suggestions are unhelpful, weird, severely incomplete or even actively harmful, and I uncheck "suggestion was helpful."

Unchecking that box is so futile that I'm not sure why I bother. Every time I improve a bad suggestion, it gets accepted while I work. It's gotten to the point where I'm discouraged from using the improve option at all.

Possible solutions (could be combined):

1. Don't act on accept/reject votes as soon as they're cast (or as soon as two agreeing votes are cast, in the case of SO). Force the suggestion to sit in the queue for n minutes.
   • If an improvement is submitted at any time, accept it immediately (like the way it works currently).
   • After the timer is up, if the total number of votes — both accept and reject — has reached some threshold, go with the majority.
   • Otherwise, add m more minutes to the timer and hope for more people to review the post or for someone to improve it.
2. If and when someone suggests improve, stop accepting accept/reject votes on that suggestion. Don't even show the suggestion in the queue anymore.
   • If the improvement is submitted, accept it immediately (like the way it works currently).
   • If the improvement is canceled, start showing it in the suggestion queue again.
   • After p minutes, if the improvement hasn't been submitted, assume the improver abandoned the attempt to fix the suggestion and start showing it in the suggestion queue again.

Another problem with the current system is the lack of feedback for participants. Suggestion reviewers have no way to know whether the community agrees with their decisions, so bad reviewers keep on obliviously doing a bad job. As long as reviewers are approving bad edits, users who suggest bad edits won't know that they need to get better, let alone want to improve, and none of the rest of this post will matter.

Suggestion submitters have it better, but only because the bar is so low. They can find out whether their edits have been accepted by revisiting the posts they edited, but only after the edits are reviewed, and who knows how long that takes? They can also check their rep, but that only tells them about approvals, not rejections, and only for up to 1000 rep, and even then only while they're under 2000 total rep. Those who repeatedly suggest bad edits can get auto-banned, but the ban isn't designed in a way that fosters improvement.

Possible solutions (could be combined):

1. Give editors more feedback about whether their suggestions are good. In the case of rejected edits, be sure to explain why, perhaps with blue-level notifications.
2. Allow the community to mark low quality suggestion approvals, and give feedback to the editor and reviewer(s). (This has been suggested before, even at this very question.)
3. Make personal review stats more prominent in user profiles. This could be a hidden section, like the current helpful flags and votes sections.
4. Add an intermediate step between "suggest all the edits you want" and "you're banned from suggesting edits for a week" that somehow tells the user "a lot of your suggestions are getting rejected for reasons X, Y and Z; you might want to reread the edit guidelines."
5. Change the reviewer requirements from being based entirely on the edit privilege to being based partially on whether the reviewer's previous accept/reject votes were correct. (Partially inspired by this idea.) ^{_{This one went through numerous iterations in my head and is probably too complicated to be implemented, but maybe it'll help with brainstorming.}} Actually, tracking reviewer quality may not be as far-fetched as I thought! Some kind of "audit" system is in the works, per dev Kevin Montrose.

Answer 2

The only thing I can think of would be to allow users to flag a specific revision when they discover a bad edit that has been approved. If enough of these flags against a specific reviewer are found valid, give them a time-out from reviewing.

I think the timeout should probably be longer for bad reviewers since it's harder to spot when it happens unless you were also reviewing the same edit at the same time. Or maybe just a lower threshold before getting banned from reviewing.

Also, the reviewer should get some sort of notification when their edit is deemed bad so hopefully they can improve before a timeout becomes necessary.

Answer 3

A very simple thing nobody seems to have mentioned so far: provide basic instructions for the reviewers. The current instructions are quite minimal (after clicking "more"):

• Approve edits you know are correct
• Reject those you know are wrong
• Improve to improve this suggested edit
• Not Sure if you are unsure and want to skip this suggested edit

What is considered correct or wrong? It's not clear. Those words could at least be linked to a FAQ, or Meta discussion. Not to mention the description of "improve", which basically says "improve to improve"!

I don't have any suggestions for new instructions right now, but I'm sure we can come up with something better (and concise), given enough discussion.

Also, I don't believe this will solve the problem completely, but it's a step forward anyway.

UPDATE: I've created a Review Guidelines faq proposal. It's still a draft, and any contribution from the community is welcome.

Answer 4

Stop tying the edit/review privilege to reputation. I'm convinced there is no correlation between the amount of reputation you have and your proficiency at editing.

You can learn to be a good editor, but only by editing a lot of questions and getting feedback on them, not by earning reputation. Someone can get the necessary reputation to review and edit without ever editing a single question.

Instead, I think the edit/review privilege should be tied to a number of approved suggested edits with a decent approved/rejected ratio. For example, say a minimum of 250 edits with a 90% approval rate.

Approving invalid edits does way more damage than just letting a few bad questions go by; it teaches new users the wrong way to edit and review.

Answer 5

It seems like most robo-reviewers accept because that's just one click, as opposed to three for rejecting. Therefore, making it take more clicks to approve a bad edit should slow down robo-reviewers.

If an edit has any reject votes, show a warning while clicking approve. It would say something like "This edit already has x reject votes. Are you sure you want to approve it?" It would have an "approve anyways" button there. Then there could be a line, and the standard reject form right there.

I'm not going to draw up an image, so here's a plaintext rendition of what it could look like:

This edit already has x reject votes. Are you sure you want to approve it?

<approve anyways>

---

Reject this edit

• spam or vandalism This edit defaces the post in order to promote a product or service, or is deliberately destructive.
• no improvement whatsoever This edit fails to make the post even a little bit easier to read, easier to find, more accurate or more accessible. Changes are either completely superfluous or actively harm readability.
• irrelevant tags This edit introduces tags that do not help to define the topic of the question. Tags should help to describe what the question is about, not just what it contains.
• clearly conflicts with author's intent This edit deviates from the original intent of the post. Even edits that must make drastic changes should strive to preserve the goals of the post's owner.
• attempt to reply [1] This edit was intended to address the author of the post and makes no sense as an edit. It should have been written as a comment or an answer.
• causes harm []

<reject>

This could also apply to other review queues.

Answer 6

One possible solution is to make it a bit harder to approve and a bit easier to reject to edits that come from a user with very high rejection ratio :

if(reviewer is OP){
    1 app./rej. gets approved/rejected //current behavior
}else{
    if(Editor has at least 10 edits){
        if(  1/10  to   0/10 rejection) 2 app. gets approved, 3 rej. gets rejected
        if(2.5/10  to   1/10 rejection) 2 app. gets approved, 2 rej. gets rejected //current behavior
        if(  4/10  to 2.5/10 rejection) 3 app. gets approved, 2 rej. gets rejected
        if(  10/10 to   4/10 rejection) 3 app. gets approved, 1 rej. gets rejected
    }else{
        2 app. gets approved, 2 rej. gets rejected //current behavior
    }
}

Numbers might change, but I think they reflect the most common types: perfect, average, needs some work and bad with their appropriated protection/handicap

Answer 7

One rather simple step would be: Do not allow to suggest a rejected edit again. I have seen that: Edits I and others had rejected were just resubmitted by the same user until they found a bad reviewer.

Solution: If an edit was rejected do not allow the suggester to edit that post/wiki again for 24 hours. That should be rather easy to implement I guess.

Answer 8

I was going to post this as a separate question until I came across this topic. If something seems off about the wording it's because it was meant as a standalone post.

I believe my point of view differs from most of the already provided answers in that it focuses more on raising awareness rather than reprisal on the offenders.

The issue I'm having with the suggested edit review queue is very simple: people are encouraging insufficient edits by approving them anyway.

I have seen a lot of edits passing by that either only partially fix something or plain deface the post that end up getting approved. The following are a few samples of edits in the last hour (2 different users), but this certainly wasn't a temporal surge of low quality edits.

Situation 1

https://stackoverflow.com/review/suggested-edits/3667680

At first glance this doesn't seem too bad. Then when you look more closely:

• Language tag in title
• Incomplete code formatting (Student and Course remain unticked)
• None of the English has been fixed

When you look at the total revisions for this answer you can see it needed 3 more revisions to get in a decent shape.

Situation 2

https://stackoverflow.com/review/suggested-edits/3667621

• Random code formatting
• No English fixed
• "Thanks" not removed

Once again a case of just clicking the {} button a few times and leaving everything else in there

Situation 3

https://stackoverflow.com/review/suggested-edits/3667617

• Language tag in title
• Random code formatting
• Error message in code format, making it hard to read

The only substantial edit isn't a particularly useful one.

Situation 4

https://stackoverflow.com/review/suggested-edits/3667520

This one actually got rejected but it still got 2 approval votes.
It literally has nothing going for it:

• Horrible title unfixed
• Random code formatting
• Added incorrect English
• Removed 2 lines of whitespace, left 3 others in there and didn't touch indentation

Still he got 2 approval votes.

Situations 5, 6 & 7

https://stackoverflow.com/review/suggested-edits/3667714
https://stackoverflow.com/review/suggested-edits/3667686
https://stackoverflow.com/review/suggested-edits/3667650

Each of these edits got approved while the only thing they do is put jsfiddle links behind a word and bolden a few words that were already distinguished by separation.

In fact there are still several (minor) issues that aren't touched at all. If these things were valid enough to let trough then the other stuff should be valid enough to reject the edit as well:

• Code formatted with bold instead of code formatting
• Irrelevant tag
• Code not indented

What am I trying to reach with this post?

I believe there are a few issues here.

• I waste time by both correcting the original post and the suggested edit.
• If it gets approved before I get my hands on it, people will read a post that has poor quality.
• A question gets pushed to the top despite nothing substantially being changed.
• Insufficient edits are indirectly encouraged because seemingly many of them are being approved.
• The trustworthy representation of a user is inflated because he gains reputation without adding value to the network.

I believe the main offenders here are the people who approve these suggestions and subsequently encourage this behaviour. We as reviewers have the possibility to stop these edits from entering the system but seemingly we're failing as a group at this task a substantial amount of time.

I won't deny that I might have set the bar higher than is commonly accepted, but I strongly doubt that these suggestions should've been allowed. Edits should ideally address all issues in a post whereas the samples I posted definitely don't do that. Either they plain deface the post and perform one specific and often incorrect action (person 1) or they invent problems and make unimportant changes.

What can we do?

I think it's important to provide feedback to people about how they voted. As far as I'm aware we don't have any possibility to see how the rest of the reviewers voted on an edit unless you explicitly keep track of the edit. This is a lot of work that will require a lot of tabs to keep open and shouldn't have to be done in the first place.

Those that suggested an invalid edit should receive a notification when their suggestion got declined. I haven't done any suggested edits in my pre-2k period so I don't if this is possible (let me know if it is), but if the "custom message" option actually gets shown to the person himself, we can use that as a way of communicating about what he should change in the future.
This would be mainly aimed at the people who go on invalid suggestion spree, usually you get a few of them if you're on a review spree.

Then again, messages might not be enough. As it is now there is a formula in place that blocks people from suggesting edits for a 7 day period if they get too many rejects.

The formula is

(rejects - (approvals / 3)) >= 5

This is in itself a good formula but I believe we can do more with it. When it comes to adjusting behaviour, going from no notion to full stop is rarely a good idea. People will get banned for too many rejections and they're basically left with one feeling: why wasn't I told in time?

I think that's exactly what should happen. As soon as you reach certain tresholds you should get a warning that you're getting too many rejections. By working in stages and positively approving people when they become more in-line with the rest of the community it's a lot easier to correct this (we'll bypass any philosophical discussions by assuming the community as a whole has the correct judgement).

By adding these warnings users will more quickly be notified when they're getting rejected a lot and will be able to improve their suggestions sooner. Likewise they will have the feedback of the positive notifications from the system when it tells them their suggestions are getting approved a lot more.

In short:

• Give us a result of our votes. Either instantly or at the end of the day let us know how the cases we ruled on were decided by the community.
• Let users know when their suggestions were rejected and why.
• Provide warnings when a user is getting close to the lockout period.
• Provide a positive notification when a user's aproval rating is rising.

Answer 9

For the record, the % of rejected, approved, and contested edits has stayed fairly static over time - but the totals have increased considerably as folks have become more comfortable with these systems. So, as with everything else, the tooling must improve along with them.

What we're doing about this:

• Short-term: increasing the number of reviews required. In practice, it seems that approving these was just too easy - more eyes on them should help.

• Medium-term: Implementing this. Better if fewer bad edits are submitted, period.

• Long-term: continue to monitor this, and listen to suggestions from y'all.

Answer 10

The low number of suggested edits being left in the queue after the launch of the beta review system is obviously not as good a sign as we may have thought, and one of the contributing factors to both that problem and the one presented here is the lack of effort required on the reviewer's part. But at the same time, the beta review system is contributing to the lack of effort. It's really a hilarious catch-22.

But the problem doesn't lie only in the fact that no effort is needed to approve an edit, but also in the fact that the users making the suggestions don't really know what they're doing.

I just saw this suggested edit, and I don't know who to blame. Do I blame the reviewers for doing an awful job, or the suggester for making an edit that detracts from the post (the URL references threw me off in the rendered view ... only the markdown view gave me an understanding of the strange situation).

So it becomes an issue of who to educate better. In the above case, I lean more towards the suggester — with better knowledge of markdown, that edit could have been pretty helpful (to me, at least; linking to documentation is always good). The intent of the edit was helpful, and from first glance, I can see why the reviewers approved it (they don't have a particularly bad approval history).

But then you have cases like this suggested edit which replaced a link to one of a YouTube video ... and got approved. In this case, I obviously lean towards the reviewers. God knows what crossed through their heads, and it was more than likely an effort towards keeping the queue clear (though I don't see why that would be beneficial to them, since there is no reward for approval).

So we're at an impasse — I don't believe there is one consistent way in which we could eliminate the problem in terms of educating users better. Plus, I think whatever systems we have in place for helping users learn the system are helpful enough.

But I'd like to bring up the argument about whether there is a correlation between high reputation and good reviewing skills. A user here argues that the correlation doesn't exist, that a user can get enough rep to approve/reject without having edited once.

This is true, sure, but chew on this: the only other way to earn reputation is by asking good questions and posting good answers (and suggesting tag wiki edits if below 5000, but +2 is inconsequential). Both of those are indicators — to me, at least — that someone knows what makes up a good question/answer, and that I could trust them approving good edits and rejecting bad ones.

But 2000 rep is definitely not a high enough threshold to determine whether a user really understands the difference between a good and bad question/answer. So my proposal would be to raise the reputation needed to review suggested edits (and, in turn, since the two privileges are associated, the reputation needed to edit freely) to 5000. At this threshold, we can at least minimize the number of bad reviewers (because 5000+ tells me that they have a pretty good question/answer quality history) and the number of bad edits approved.

(And in case anyone brings it up, I'm pretty sure this user, who approved the spam YouTube edit, is either compromised or did it purely for kicks...the world may never know).

Answer 11

People make mistakes. You can't avoid that. The suggested edit interface is easy to use. But unfortunately, it is easier to accept than to reject (you have to give a reject reason). And when in doubt people tend to the positive side (at least the helpful ones).

Besides, if a post already has a accept or decline vote, it is psychologically attractive to give the second vote: clears the queue with one.

But as I look at the size of the edit queue, I don't think there are much people active on this. So maybe some education can be helpful. Maybe a small e-course for reviewers.

Answer 12

In addition to anything else, or as a separate measure, I think that one way to prevent bad edits from being accepted would be to allow the person who made the original post to approve or reject the edit before anyone else has the chance. If they don't do anything after a certain length of time has passed, or if they actively assert that they don't want to go either way, put the edit into a review queue to go through whatever other process is in place. I think that this would work because the original poster is more likely to understand and care whether the suggested edit is an improvement, also it gives them a more direct chance to learn about how questions or answers should be articulated and formatted.

Just to clarify, here is the workflow I have in mind:

1. User A posts a question or an answer.
2. User B decides to edit the post made by user A
3. User C comes along and decides to edit the post, but cannot because User A has not done anything yet
4. User A has the opportunity to Accept, Reject or Skip the edit
   • If User A accepts the edit, it is applied to the post
   • If User A rejects the edit, the changes are rolled back and the post can be edited by User C
   • If User A chooses to skip the edit, the post goes into a review queue
   • If User A does nothing after a certain length of time, the post goes into a review queue

Answer 13

I actually signed into meta to answer this. Here is what can be done:

After an edit is peer viewed and approved, let the edit be notified to the original user and ask if he would accept this as an edit and the reason for edit.

Ofc. User whose post is edited must have some reputation to get this privilege.

Every edit which is not accepted by the original poster has to result in edit going back for peer view (hopefully, this time someone else) and depending on that result, reputation has to be negated from the first editor.

Confused?

Again...

ABC posts a question, PQR edits it, if ABC has reputation of 100 (say for example) he can see that his post is edited with the reason for edit and ask if he likes the suggested edit, if he likes it, edit was good! and has been approved by an upcoming stack member, if it is rejected or disliked by ABC only then, this time, edit goes to XYZ. If XYZ approved that PQR's edit was good, PQR gets +rep else -rep.

What will be achieved by this?

1. Member who edit will have feedback on HOW NOT TO edit rather HOW TO edit.
2. Member who got his post edited and perhaps liked the edit will know how to post it in future. At the moment, when I was new (see my stackoverflow profile, I have joined recently) Someone edited my post and I kept wondering as to what was changed!!! Trust me, I still do not know! However, learnt it across on how to post keeping everyone in mind, even those who seek answers on search engines. If you could see I have edited few posts where members rep was way more than mine and also has longer history than mine, yet, the sense of posting the content has not been authored in their habits.

Answered, not for bounty but, to put things across. Good that you asked! :)

Answer 14

I've seen perfectly valid edits sitting with a single reject vote, so the inverse to your question is a part of the same issue -- What can we do to stop good edits from getting rejected?

If you can find examples of bad edits accepted, it stands to reason that there are good edits being rejected (in equal numbers?) as well, and are thus gone from our purview. Can one of the data wizards pull stats? What percentage of edits are accepted vs. rejected?

It seems to me that the more alarming of the issues you present are pertaining to questions or answers. Tags, being more visible to experienced users (because we use them on our own questions) will self-correct over time, IMO. A question could be silently polluted by a bad edit (or suffer for lack of a rejected good edit) and stay that way until someone "in the know" happens across it.

I have two suggestions, both involve using systems that are already in place to make sure that a set of qualified eyes are on an edit before a decision is made:

Suggested fix #1 (most restrictive) only show body edits for questions or answers for which the user has at least a bronze badge. This makes sense -- until I've received 100 upvotes on a given tag, what is the likelihood that I can accurately evaluate the quality of an edit to a question or answer on that subject?

We can continue to show tag wiki edits and question tag changes to anyone with review permission, but I think reviewing the -content- edits should be reserved to those who are demonstrably equipped to evaluate them.

Suggested fix #2 (less restrictive) 2 reject/accept votes required when at least one of those votes comes from a user with a bronze badge in the given tag, OR 4 reject/accept votes from all non-badged users.

Again, we can continue to show tag wiki edits and question tag changes to anyone with review permission, but we either require more "unqualified" votes or a "supported qualified" vote to change the body or questions or answers.