The robo-approvers are killing my will to review edits

Question

Already I have lost all hope of looking at late answers etc. It's just a parade of meh. I can't make any kind of intelligent decision because I can't see the other answers to the question, for instance. There's no Looks Good, only Skip, and those feel different to me. So I've been more interested in reviewing edits.

But people keep approving crap while I'm looking at it. Two or three "the post was already approved; please visit to edit" and I'm just "screw this whole thing".

Can we do something like one reject bounces all existing votes: you need three approves in a row? Because a lot of people are stupidly approving and actually driving me away from the queue.

---

_{Note this is less about letting bad edits through and more about keeping reviewers motivated.}

Answer 1

The best solution I've found to this problem is to not review posts from a queue.

Some of the other ways that I've found to find content to review are:

1. Participating in tag cleanups. It's not just about removing tags from posts; it's about fixing all of the problems with a post. Since there are no queues for this, it's all voluntary, so most of the other people doing this with you will be enthusiastic and capable reviewers.
2. Just scan through old posts. There's enough things that need fixing that random searches are reasonably productive.
3. Query posts based on criteria likely to produce content needing editing. Sort by descending votes, look up tags that tend to produce low quality questions/answers (i.e. homework), find/create complex custom queries designed to return low quality posts (example), etc.
4. Look through the questions of people posting on meta with "I can't ask questions anymore, WTF!?" (They tend to have lots of content that needs reviewing.)
5. Before answering questions in general, or immediately after answering any question, edit the questions as appropriate.

I stopped using the queues weeks ago, but I won't let that stop me from reviewing content anyway.

When I do go to the queues to review (which is less and less often every week) it's usually to find problematic reviews that need fixing. I look for edits that shouldn't be approved, favorite the link, and then comment/rollback the edit when it's approved, for example. Or, if the reviews tend to be just incomplete, not abusive, I just improve posts (now that the race conditions of doing so are lessened). I may not get review points for it, but the edits still take effect, which is enough for me.

Answer 2

people keep approving crap while I'm looking at it. Two or three "the post was already approved; please visit to edit" and I'm just "screw this whole thing".

Above is pretty frustrating in my experience. It is especially painful in cases of blatantly obvious crap that takes me just few seconds to reject.

It would be interesting to try exclusive review period to remedy this. By this, I mean that picked suggested edit is taken off the queue for 2-3 minutes so that no one else can review it until timeout expires.

This would guarantee that my decisions (at least reasonably quick ones) could only clash with those of the user(s) who held the suggested edit longer than mentioned timeout - which would weed out mindless click-through robo approvers.

---

Yet another way to tame the problem could be to increase delay for review action buttons.

• Current delay is 2 seconds. Assuming that thorough reviewer spends one minute on analyzing and improving the edit, this means that robo-approver is 30 (thirty) times more "performant".

Think of it... 3 (three) rubber-stampers acting in parallel are capable of blindly approving 30 edits a minute, potentially destroying efforts of 30 (thirty) responsible reviewers who could have been spending this minute working on mentioned edits.

Increasing action buttons delay to, say, 20-30 seconds could help in somehow leveling playing field in favor of "good guys".

---

_{For the sake of completeness, I am aware of various ideas to detect and block fake reviewers (listed eg in linked questions) but here, I focus exclusively on how to block them from messing with my review process.}

---

update Dec 20 '12

A new feature has been introduced that will likely help responsible reviewers part ways with robo-clickers:

An optimization to the Late Answers, First Posts, and Low Quality review queues has been deployed that will keep a single review from being shown to multiple people at the same time...

---

update Aug 31 '14

Suggested edits review queue now works so that

...each person reviewing suggested edits has a good opportunity to accurately review them.

It's a bit like ticketmaster, or any type of reservation system. When you visit a suggested edit review task it's now "checked out" to you for that time - the counter previously didn't reflect the amount of "checked out" tasks.

Answer 3

Make better edit review audits. The current are terribly bad, for example 'vandalism' is imitated for inserting some random words, so obvious, you could as well put big text THIS IS AUDIT withing the text.

Make some more finesse audits. For example change some link to something posting to some off-topic page (or to tinyURL-like redirection to the image with information this is audit only). This will check if people notice that someone has changed link and if they actually click the link to check what's behind.

Make some audits changing closing tags for XML, removing dots from method calls, changing math operators or removing spaces from python code. Check if people actually analyse such 'improvements' in code.

Change 'I' to 'i', 'you' to 'U', or add 'Thanx for help!', 'Urgent!' or similar. Check if people notice and reject.

There are many things you can do with edit review audits. As for now, it's the poorly audited review queue.

Answer 4

They could just track the user who went into the question to edit it, make sure the button is enabled for them even if it's already been approved, and give them credit for the review.

Answer 5

This is still a problem. If changes were made to the site in late 2012, they did not solve it.

The solution seems obvious to me. If you have taken the effort not only to review edits, but also to enter a post to manually improve it, you are obviously more seriously concerned about the quality of the post than the robots are. So when you enter a post to manually edit, it should be locked for approving/rejecting by others for 10 minutes or so.

Perhaps with the exception of the original poster themselves and moderators.

Answer 6

As part of a larger set of changes to suggested edits, we've attempted to address this bit of frustration by adding timed exclusivity for reviewers:

When you view a pending suggested edit, the system will avoid assigning that edit to any other reviewers until you've submitted your review or a reasonable period of time (currently 3 minutes) has passed. The number of "in review" tasks is tracked, and the main /review page updates the counts accordingly.

This is... considerably less dramatic than your suggestion, but it's much easier to calculate and explain, and so far appears to be quite effective.

Answer 7

Limit the Number of Reviews non-Moderators can do to 1 per X Seconds

Limit the number of reviews a reviewer can perform to one review per {amount of time between 10 and 60 seconds}. Don't enforce an on-page penalty, but an overall penalty and apply it to all types of reviews. This will at least give anyone a chance to grab a review. I can't add a comment until 15 seconds pass, this is the same sort of "penalty" that would only truly affect automotons.

Mechanics

The {10-60} second counter starts counting down when the review is presented to the reviewer. The review itself can take less than 10-60 seconds, and the review action can be submitted at any time, but the reviewer will be unable to access any item to review from any queue until that counter hits zero.

The time limit on the counter could be based on the type of review presented to the user based on an estimate gauged on the overall average time needed to process the review type. A First Posts presented could have a longer counter applied to it than, say a Low Quality Post or Suggested Edit.

Real reviewers won't feel the penalty because they should actually be taking the time to review.

Even if this doesn't stop the robo-reviewers, it will likely allow more reviews to be available for non-robo-reviewers to access and provide some atmosphere of fairness.

There is also the potential to reduce the penalty based on reputation. 60 seconds may be too much time, but 10 is (in my opinion) not enough time; it can be 30 seconds or 45 seconds, but it must be some amount that's semi-substantial.

Answer 8

Even though we have an accepted answer, here's an idea:

Penalize the non-conformists

• Every Suggested Edit review requires "approximately" 3 agreements from reviewers (right?).
• Users get 20 Suggested Edit reviews daily

I recommend the following after a reviewer reviews a Suggested Edit:

• Reduce their the daily Suggested Edit review quota by 1 as normal.
• Delay assigning points towards their badge progress until a decision is made about whether the Suggested Edit is accepted or rejected.
• if the reviewer's Accept / Reject decision agrees with the actual result of the overall review, then count that review towards badge progress.

Certainly there will be those who are illegitimately penalized by this, but of all the queues besides Close Votes, Suggested Edits is the most active and any perceived loss in badge progress would be eventually re-gained. And real users shouldn't care as much about a minor loss of badge progress if it helps prevent abusers.