Suggested edit system is causing grief, needs to be rethought

Question

Over the past few weeks, there have been a number of complaints about users making suggested edits en masse whose quality was at least questionable.

Let an example of a low-quality edit be the addition of the web-frameworks tag to dozens of questions about the PHP web framework Laravel. No one is ever going to use it in a search, but Laravel is a framework, right?

(If you disagree about this being a low-quality edit, please bear with me anyway - my point here is that the system through which such conflicts could be settled is broken.)

Examples

• Where do we draw the line on esoteric tag additions?

• tags of mundanity that are gratuitously applied to posts

• https://meta.stackexchange.com/questions/220847/mass-tagging-of-laravel-questions-with-web-frameworks-how-to-react

• Yet another case of serial minor tag editing

• More serial minor edits

This is a toxic situation guaranteed to create strife because -

The new user is happy to have found a way to participate and earn some rep; they may be genuinely feeling that they are improving the site. The reviewers greenlighting most of their suggestions confirms them in their belief.

The reviewers usually see only one edit - and it's easy to see how they might go, "meh, this isn't a terribly useful edit, but hey, maybe it helps someone, and it's a new user. No point discouraging them."

The veteran users who check out the editor see dozens of what they perceive as low-quality edits, usually ignoring anything else that should be fixed in the post. To the veterans, the new user is systematically looking for excuses to make their pointless edits, creating a giant mess in order to gain some rep - a mess they have to clean up, while there are no consequences for the editor, who gets to keep their ill-gotten points. While a bad question or answer can be downvoted - usually leading to a change of behaviour eventually - the edit system looks like it's wide open for gaming.

So tensions are high, and when there's dialog it can be angry. I'm not saying either side is completely right. I'm completely prepared to believe that many of these edits are in good faith, etc. But it's a mess because it's frustrating people. As said, there are no consequences for bad-faith mass editing; apart from commenting on some random post, there is no way to communicate with an editor. The outcome of a moderator flag is very much uncertain. The absence of a clear set of rules means there's plenty of room for heated arguments.

Something needs to change here.

Either

• We need super clear rules on what makes a good tag edit

• Tag-only edits should be allowed only for 2k+ users, or they should not gain 2 reputation points for <2k users

• The possible 1000 rep gain for suggested edits needs to be rate-limited (say, 200 per month) (this is my favourite!)

• The possible 1000 rep gain for suggested edits needs to be lowered, say to 500 or 250

• Rollbacks of suggested edits should result in annulment of the 2 points gained (this is terrible)

• There should be a system to vote on edits (yes, that's been suggested before)

• There should be a defined and officially sanctioned pathway, and ruleset, for moderator flags cast about these users

... or something else. But something needs to give.

Answer 1

Bad edits aren't limited to suggested edits. We tend not to notice the bad non-suggested edits because they don't get a minimum of two reviewers involved (three on SO).

One thing that we're very bad at is teaching what is a good edit and what is a bad edit. I've reviewed a lot of suggested edits in my time. I'm not in people's mind, of course, but I believe that in a majority of cases, the editors thought they were doing a good thing. Not all the time — I've also seen clear badge hunters — but often the editors gave me the impression that they were attempting to improve the site.

Even when I'd managed to reject a suggested edit in time and left some specific feedback such as “Please do not use code markup for proper names, only for code” or “Wikipedia content must be attributed and does not make an appropriate tag wiki in 99% of the cases”, it seemed a bit pointless. The editor is, practically speaking, never going to see that feedback.

An absolutely necessary change to improve the quality of edits is to notify editors of the reasons for rejected edits.

A lot of these people would be willing to learn, but we aren't seriously trying to teach them!

When bad editors become reviewers, they'll keep applying what they're used to, so the system tends to perpetuate itself. Merely teaching editors would start a virtuous circle which I would expect to drastically improve reviews as well, after a delay.

An argument in favor of this virtuous circle theory is that I've observed far more bad suggested edits on Stack Overflow than on other sites (tag wikis excluded).

If this isn't enough — but let's first do it and see the results after 6­–8 weeks — then we should teach reviewers as well — and editors with the editing privilege, for that matter. This could take the form of improved audits for suggested edits. Right now, we throw garbage as people and make sure they're rejecting it as such, which catches robo-reviewers, but not incompetent but well-meaning reviewers. We should have audits that reflect the actual typical kinds of good and bad edits that the sites see. (I'm sure this has been raised before on MSO.)

Discussions on MSO show that many people who review edits have never read the editing guidelines, or didn't really pay attention to them. Since suggested edit review doesn't lack manpower, we could make becoming a reviewer more difficult, requiring passing several audits (drawn from real-life cases) before getting access to real posts. This could be done on Stack Overflow as a stopgap measure while attempting to shift the balance after starting to educate suggesting editors, in order to ensure that the feedback that editors get is correct.

Answer 2

This has been my pet peeve for quite some time and I'm absolutely bustin' to do something about it.

From my totally unscientific casual observation the majority of the bad edit suggestions that I have checked into are made by people with under 1000 rep, and the approvals are usually done by users with under 5K rep, mostly in the 2K-4K range.

This annoys me a lot, because the approvers should know better. It's not the fault of the people making the suggestions - bad suggestions from well meaning people and worthless suggestions from people trying to work the system is an expected behaviour that the process should accommodate.

Taking into account my observations mentioned above, it's obvious to me that the reviewers need to be reviewed or audited. My suggestions on how to effect this are:

• establish a new Review Audit queue. This queue is accessible only by trusted users or those who have already accumulated considerable editing experience, i.e. the Copy Editor badge¹.
• the current edit suggestion mechanism remains the same. We know there will be bad suggestions, we're cool with that.
• any suggested edits that are approved soley by reviewers with <5K rep, or are approved by a margin of 3-2 are then sent to the Review Audit queue. Anything that appears in this queue has the original reviewers anonymised out - it doesn't matter to the auditor who the original reviewers are

• items that appear in this queue have an Approve button, an Undo button, and an Improve button, just like the existing suggested edit review queue

  • the Undo button rolls back the edit approval, reverses any rep or incentive that was allocated, and optionally penalises a rep point or two from the original reviewers. As the original reviewers have been anonymised from the review there should be little or no chance for malicious reversal. If a small penalty is applied to the original reviewers it will hopefully make them think a little more carefully, as most users in that lower rep range still cherish every rep point they have. When a reviewer has been revoked a specific number of times (or reached a certain revoke/approve percentage) then their review capability can be automatically suspended for a period of time.

  • the Improve button does what the existing improve button does, except it also removes any incentive the original reviewers got (i.e. reduces their review aggregate towards certain badges) but doesn't penalise them in any other way. Along with the already existing "This suggestion was helpful" checkbox good edits that were insufficiently thorough can be improved.

So this is a rough suggestion of a workflow that could have a positive impact on the problem and shouldn't take too much to implement. Amounts and numbers are suggestions and open for debate. The entire workflow is open for debate. Please discuss - this problem isn't going away anytime soon.

¹ Note that I didn't nominate the Steward badge, as getting this doesn't necessarily involve gaining much editing experience

Answer 3

<2k users submit edit suggestions, and get +2 for every approved edit, up to 1000 rep. This enforces to new users that improving the site is encouraged.

It also is ripe for abuse.^{citation not needed} As noted by OP, an edit (and 2 points) can be made in 20 seconds, for a rate of 360 rep per hour, or the 1000 limit in a mere 2 hours 47 minutes of work.

Stop and think about that for a minute.

3 hours for 1000 rep. I've worked my way up the ranks quite quickly this year, but 1000 in 3 hours is insane.

Reputation is a rough measurement of how much the community trusts you

Think about how much you trust a 1000 rep user. Now think about if that were earned in 3 hours. Without a single question or answer.

To me, the solution is clear:

Limit edit rep gain to +20 per day

That's 10 edits. Meaning it now takes 50 days to get that 1000 rep. That's about as long as I've been a member, despite my 8,216 current SO rep.

It is almost always better to remove the incentive for abuse than to try and impose artificial restrictions that penalize valuable contributions.

Don't limit the rate of edits

We don't limit answers just because you've hit your rep cap. The +2 token rep reinforces that improving the site should be encouraged. Becoming an "established user" who's "been around for a while" through a few hours of edits shouldn't be.

Answer 4

I think a rate limit may be reasonable...

Its been suggested before, but limiting edits to X reviews per hour would help a lot.

Perhaps we could discourage spree editing even more by limiting low rep users to 1 edit every 3-5 minutes, it would encourage new users to edit more organically, and allow reviewers to easily keep up with problem editors.

Answer 5

Rollback Penalty

Rollbacks of suggested edits should result in annulment of the 2 points gained

Not sure of the reasons for rollbacks, but if because of bad edits, then why not remove their "undeserved" rep?
Or, more?

It's great, and nice - hugs etc, when users help out. But they need to be stopped when they're not helping out and are causing other user time to be wasted which could have been spent actually helping out.

For the rollbacks, something like:

2 rollbacks in a week;
AND/OR
4 rollbacks in a month;

= -5 rep

8 rollbacks = 1 month edit ban
3 bans = edit ban for life

(Note: I don't know how frequent rollbacks are, so perhaps totals of 2 and 4 might be inappropriate).

And similar system for rejections (suggested further below)

---

Changing Rep Gain

1000 rep gain for suggested edits rate-limited (say, 200 per month)
1000 rep gain for suggested edits lowered, say to 500 or 250

Tighter limits doesn't solve the problem of poor edits, it only reduces the frequency and ultimately total bad edits per user.

So while these suggestions are arguably an improvement, they're not the resolve, and they're also potentially detrimental as users carrying out valuable edits (who are also in it for the money) are also limited to the monthly frequency and over total of good edits they will do.

It smells to me rather like a typical (government-like) blanket policy which "partly resolves the issue while punishing others who don't deserve it, and also introduces another issue".

---

Penalties for Rejected Edits

Blanket policy isn't ideal, and getting reviewers to do more work is unfair and likely result in less reviews.

The solution is simply harsher punishment for users who repeatedly suggest poor edits.
Whether they're just after rep, or were genuinely trying to help, they need to be informed, and the problem stopped.

The key to stopping it is "per bad user", and I know edit bans are in place, but they need to be tighter, with rep loss!

I know some will argue against rep loss, but bad reviewers often do it for the rep, so hit them where it hurts!

So, review queue for edits. People either vote:

1. Allow
2. Reject
3. Terrible (new possible option)

So reviewers vote from the above then punishment can be dealt with as follows:

Penalty System

Each criteria, that is 1 OR 2, gives you 1 month edit bad and -5 rep.

Users with Terrible:

1. 3 Terrible in a week
2. 6 Terrible in a month

Users with Reject:

1. 4 Reject in a week
2. 8 Reject in a month

Users with Terrible and Reject:

1. 2 of each in a week (4 total, any mixture)
2. 3 of each in a month (6 total, any mixture)

(Sure Stack/others could poke some holes in the above numbers, but you get the idea)

For all above:
Users with 3 bans in any 6 months period, OR total of 5 bans any period, are banned from editing for life.

I think after 2 bans they should have read/asked what they're doing wrong!
And if not, then they're harming the site and wasting other users time and not likely to resolve it.

Remember, we're dealing with people who just don't care. Laugh at bans, and will edit badly and quickly just for some quick rep points/hope to remove their question asking ban!

Terrible and Reject are both "no", so reviewers who vote mixture of both still decline a suggested edit, however it introduces a fair voting towards potential ban.

For a user to get a "terrible" mark on their total towards a ban, has to have 3 terrible in one review.

To get a rejected added to their total rejected, they just need a "no" (mixture of rejected and terrible).

---

They Should Defo Be Notified

For rollbacks and rejected edits [current system], they need to know what they're doing is not helping, so a message in notifications:

Your edit was reversed/rejected as it was cack. Please read (link) help section page of how to do a good edit and avoid rep loss

This gives genuine editors a chance to review why and redeem themselves (we've all had some rejected..).

Answer 6

I agree with the idea that a rollback of your edit should result in loss of any reputation gain from that edit, and in addition an automatic notification should be sent.

In addition, I propose limiting an individual to getting 2 points of reputation per edited post. As it is, someone can suggest numerous trivial edits to a post, instead of making all the edits at once.

Here is an example where I rolled back two edits, one invalid and one trivial (the rollback was due to the invalid edit; the trivial edit was subsequent), and the same user subsequently proposed three more edits to the post (all accepted, of course): https://stackoverflow.com/posts/21813938/revisions

Answer 7

A few thoughts…

Clearly the editor should be told when their edit is rejected or approved. If an edit is approved by less than 100% of the reviewers then the editor should be told way some people did not approve it even when the edit was approved. This will all lead to leaning.

Removing -8 rep from a editor that gets there edit rejected may speed up the learning, maybe limit the rep removal to the rep gained by user from edits over the preceding few days.

On Stackoverflow the edit review queue is normally empty. We have people power to play with….

I expect that slow reviewer do a better job, but yet the current system stops them reviewing more than 1 edit, as the edit queue is empty again by the time they have reviewed the first item.

We don’t have to allow all reviewers to review all task, so for example we could require each edit to be reviewed by at least 1 person that has a history of rejecting a lot of edits, or at least 1 high rep user etc. (Or just leave the item in the queue until at least one known “good” reviewer has approved it.)

Edits that get a reject vote should need a lot more approve votes to get approved or approved votes from more trusted users.

Maybe an edit should be in the queue for at least 5 minutes regardless of the number of votes it has already got.

When reviewing edits if there are multiply edits from the same user in the queue, they should be shown to the same reviewer. The reviewer should have a way to reject edit they have just approved if they see a pattern of bad edits…

Maybe if a user has done lots of edits in the last hour, there edits should remain in the queue for longer, so they are more likely to be reviewed by the same person.

Maybe new reviewers should have any “approve” votes they cast ignored until the system can see that they are rejecting edits that get rejected by trusted reviewers.

Any reviewer that approves more than the expected number of edits that get rejected should be shown these edits and the reason why other approvers rejected them, so they can learn.

Any reviewers that approve a lot of edits that get rejected should be flag for the mods to investigate them.

Answer 8

I think earning rep is fine for the first 500 edits or till they hit 2000 reputation total. It gives new users a chance to earn decent reputation if they are still new to programming. It enables them to offer a decent bounty for one of their questions that they really need help with. The problem with bad suggested edits is a problem because of bad reviewers, those people with over 2000 rep.

I've seen people changing only minor edits every time before, and I would reject all their edits for being "too minor". I haven't seen those people in the queue again because they likely got banned from suggesting edits. There is no reason to ban reputation gained through edits for new users/low rep users just because of some bad apples.

As an example... lets say that we just ban questions from people with under 50 rep, because people with under 50 rep are more likely to get their questions closed and downvoted! You see.. it's not fair to ban questions from new users just because of all the bad question askers.

I would be in favor of an automated system that does NOT reward any reputation for suggested edits that only add 1 or 2 tags... since most posts have something else that can be changed, and only adding 1 tag, it's likely the person is just doing it for reputation and doesn't really care about helping the quality of the site.

The possible 1000 rep gain for suggested edits needs to be rate-limited (say, 200 per month) (this is my favourite!)

This is like my least favourite! I've only been a SO member for like 3 months now, and I hit my 500 edits in about 3 weeks. People on this site know how to use computers and know how to type quickly, and multi-task quickly. I've noticed that one of the reasons Stack Overflow is so popular is because they understand this! I see examples of this implementation all throughout the site.

The possible 1000 rep gain for suggested edits needs to be lowered, say to 500 or 250

I would have no problem with it being lowered to 500. But 250 is too low.

There should be a system to vote on edits (yes, that's been suggested before)

We already have that, it's called the suggested edit's review queue :) , and then we already have roll-backs, so the person's whose post was edited gets notified each time an edit was made to their post. They are the person who has the final say really.. If it's not a good edit, they will roll it back. If say that member is no longer active.. well, whoever views the question can roll it back or do another edit. If the question is very unpopular view wise... well, nothing we can really do about it.

We need super clear rules on what makes a good tag edit

Yes, for sure, this would help educate reviewers.

---

EDIT :

I was reading some of slugster's answer, and that reminded me of something else.

Yes, the suggested edit review queue AUDITS need to be IMPROVED GREATLY.

I have NEVER seen a suggested review audit that the correct answer was to "ACCEPT" the edit. It is always to "REJECT" the edit.

Also, I think the easiest audits to pass are in this queue too. I guess the text is like randomly generated from other posts that were deleted/rejected or something... There needs to be more time spent on creating good AUDITS that are hard to pass in the queue.

For example, some audits could change only the spacing of the code , that wasn't bad to begin with... but if someone approves that edit, FAIL THEM..

---

EDIT 00000002 :

I think the best way is to send a notification to the users inbox each time a suggested edit gets rejected. And then if they click on that it should bring them to the part in their profile showing that it was rejected and why it was. The reason people don't know that they aren't editing correctly is because they don't know how to check it normally until they already have like over 300 edits or so. I think it's important enough to send these messages each time, just like you get notified everytime someone edits one of your posts.

Answer 9

One possible means to improving the quality of reviews is to add the reputations of the reviewers and require over (say) 20000 reputation points before a decision is made. This idea is based on a comment elsewhere in this Q&A suggesting that some low reputation are approving too many poor edits.

I am new at reviewing suggested edits and already I have seen several edits approved that I thought were wrong. One problem I see is that a few people quickly approve the edit before I can finish typing a comment. Sometimes others reject for a similar reason to mine, but jointly we were too slow. So slowing down the review process for an specific edit review seems useful.

Answer 10

Reps gained from edits, should not count toward being allowed to review edits.

So if someone gets lots of rep from miner edits, at least they can’t then go on and approve other peoples miner edits.