-10

I understand that this may be quite subjective.

Please direct your attention to Exhibit A. The username has been blurred to use as an example.

You may or may not agree that this edit is too minor. My concern is with the ratio of approved to rejected edits for this particular user and further, the entire community.

Sample set of edit numbers for 10k+ users:

╔══════════════════╦══════════╦══════════╦══════════════╗
║                  ║ Approved ║ Rejected ║ Reject Ratio ║
╠══════════════════╬══════════╬══════════╬══════════════╣
║ Conrad Frix      ║      588 ║      377 ║ 39.07%       ║
║ Gajotres         ║      576 ║      217 ║ 27.36%       ║
║ Ivaylo Strandjev ║      185 ║      144 ║ 43.77%       ║
║ ughoavgfhw       ║      257 ║      250 ║ 49.31%       ║
║ FreshPrinceOfSO  ║      189 ║      540 ║ 74.01%       ║
║ ---------------  ║          ║          ║              ║
║ Exhibit A User   ║      652 ║       35 ║ 5.1%         ║
╚══════════════════╩══════════╩══════════╩══════════════╝

I think that a minimum reject ratio should be enforced (I would be very curious to find out what the actual ratio is).

Possible contributing factors

  1. Lack of training on what to look for
  2. No negative consequences for approving a poor edit
  3. User being simply too nice

Possible solutions

  1. Training on suggested edits. Have a "course" of 20 or so edits to test users on competency. Encourage the use of button.
  2. Notify user their reject ratio is too low
  3. If there are too many edits that are "rejected" and user approved, prohibit user from reviewing for x amount of days.

See also

|improve this question
  • 14
    I would rather find a way to make reviewers review appropriately. And I don't think that some enforced ratio is the way to do that. It might be an indicator for some extra attention, but enforcing it might lead to all sorts of nasty side-effects. – Bart Apr 8 '13 at 15:25
  • 4
    What about a user who would skip all the ones that he thinks needs to be rejected and approve the one that he knows are good. – Hugo Dozois Apr 8 '13 at 15:26
  • 3
    Also, what about a too high reject-rate? – Daniel Fischer Apr 8 '13 at 15:26
  • 1
    @DanielFischer Probably a case of being too mean of a human being and needs social training. – Kermit Apr 8 '13 at 15:27
  • @Bart I'm all for some sort of training. Whatever improves the quality of the edits. – Kermit Apr 8 '13 at 15:28
  • 3
    @HugoDozois The trend that I'm noticing is that too many bad edits get approved. I wouldn't be concerned about edits being rejected. – Kermit Apr 8 '13 at 15:29
  • 2
    @FreshPrinceOfSO That's not what I meant. I'm talking about someone who would have a high accept ratio because HE skips the bad edits/ the ones he aren't sure and only review the very good ones. I know this isn't the best thing, but still it would be a possibility. I tend to agree with Bart, we should educate people instead of restricting them to a ratio. – Hugo Dozois Apr 8 '13 at 15:35
  • 2
    I think we should have a balanced ratio by measuring the deviation. If they deviate too much in one direction, then the amount of review audits should be dialed up. – Brian Apr 8 '13 at 15:36
  • Maybe we should add audits to the tag review system? – Cole Johnson Apr 8 '13 at 17:39
  • 1
    Do we have an [external] tag now? Kill it with fire. – PeeHaa Apr 8 '13 at 18:11
14

Looking at rejection ratio is not a reliable way of determining the quality of a reviewer. If a person's rejection ratio deviates significantly (in either direction) from the community average's reject ratio they can still be a good reviewer who makes good decisions. Likewise, someone who has a reject ratio right in line with the community average can be a poor reviewer who frequently makes poor decisions.

This fundamentally comes down to the fact that people can skip reviews, and are encouraged to do so freely. People frequently end up creating a niche of types of reviews they like to focus on, and skip others. Since some of these niches can have very different accept rates, this can greatly skew the reject ratio of even very good reviewers.

While I'll be the first to agree that there are problems with the review system, despite all of the effort that has gone into improving it, this is not a good path to go down towards encouraging quality reviews. Remember that goal; we're looking to get quality reviews, not just more rejections.

|improve this answer
  • The point about niche skew is big. If someone goes through the queue and only handles the ones they feel comfortable in, and that has a higher- or lower-than-average rejection rate (maybe it's very complicated, and edits are often wrong giving an abnormally high reject rate, or it's very elementary and mistakes of new users are often corrected, giving an abnormally low reject rate), you would also expect the user to carry that trend. I was actually brought here because I noticed my reject rate was abnormally low and was researching the matter. – corsiKa Jul 4 '13 at 16:16
4

The point @Bart makes in his comment ...And I don't think that some enforced ratio is the way to do that... is spot on. There is a level of unfairness is defining an arbitrary value and using it to gauge the quality of a person's work, especially when they have limited control of the input. It is conceivable (although unlikely), that someone reviews a higher than normal percentage of quality edits. To @HugoDozois's point, it could also be a situation where someone is skipping poor edits rather than rejecting them.

Given this, I don't think any automatic punishment is appropriate, so I would not support your 3rd solution.

However, training and/or some extra attention might be warranted.

I think it has been suggested before, but I can't find a link right now. I believe we should have another review queue for reviewing reviewers. Maybe make it part of the 10K/mod tools, or just a standard review queue for 5K users.

The purpose of the queue would not be to review the quality of the edit, but to review the quality of the approval/rejection.

  • If a user drops below a specific reject ratio, then all reviews are subject to peer review
  • The exact ratio should not be made public due to the potential for "walking the line", or adjust the ratio based on the community average.
  • Have multiple reviewers responsible for validating the review and vote on if a particular review was good, bad, or borderline. I think a "borderline" vote is necessary because there is so much gray area for approving edits.
  • If a user gets too many "bad edit" or "borderline edit" votes, then they should be subject to review suspension
  • Review suspension length should be tiered based on the number of review suspensions. First offense is a warning and required "training". Second offense should be 5 days with additional suspensions increasing by 5 days each time until they get to 5 suspensions. At 5 suspensions, their review privileges should be subject to a lifetime ban (maybe with a chance for parole).
|improve this answer
  • Please don't turn this question into a global "place to suggest things to fix the review system". We already have several of those, and one primary one in which this exact suggestion has been made many times. If you want to discuss a review queue for reviews then find one of the primary past discussions on it and see if you have anything to add (keep in mind there are a lot of variations on it, so read around before posting). – Servy Apr 8 '13 at 15:52
  • @Servy sorry that wasn't my intention to do that. I honestly was just trying to response to the OP's idea that the reject ratio was a measure of the quality of the reviewer and provide an alternative solution to what the OP proposed in the question – psubsee2003 Apr 8 '13 at 15:55
  • I should point out that moderators do now have tools to evaluate reviewers in detail, and to act on reviewers that don't seem to be paying attention. For people above a certain number of reviews, now that the audits have dealt with the worst offenders, it's not easy to find truly bad reviewers on that list. – Brad Larson Apr 8 '13 at 16:00
2

I don't know that enforcing a specific ratio is the right way to approach this. I've seen very responsible reviewers with relatively low rejection ratios, I believe because they skip over many edits that others could have rejected.

The suggested edit audits appear to be reducing the number of rubber-stamp reviewers, and we have recently been given tools to examine reviewers and apply manual review bans when needed. I do regular sweeps through there to look for troubling review behavior. Yes, one of the first things I do is sort by acceptance percentage, but even there I'm finding it harder and harder to identify people outright abusing the review system.

I think we're now getting into more subjective territory as far as what should or should not be rejected. How minor is too minor for an otherwise valid edit doesn't seem like something that would be easy to enforce system-wide.

|improve this answer
1

This problem has already been addressed with review audits, and moderators can examine the reviewing patterns of any user. Imposing a minimum rejection requirement would interfere with the task of reviewing suggested edits, by forcing reviewers to skip or reject perfectly valid edits if there are too few invalid edits.

For reference, as of writing on Super User:

DragonLord has approved 92 edit suggestions and rejected 23 edit suggestions

|improve this answer
-1

I would favor a 80/20 split. This is derived from the Pareto Principle. Looking at my current ratio, I have an almost exact 70/30 split. I would expect this because I am a stickler when it comes to minor edits and invalid edits. The system could be tweaked to turn up review audits on persons who skew too much in one direction or too far from the Pareto factor. This should deter extreme examples of too many approvals or rejections because the system will begin to put people in timeout and all out ban them eventually.

|improve this answer
  • 2
    I strongly disagree with the notion of a single perfect ration that could (or should) be enforced. Mostly based on the reasoning @Servy posted here. – Joachim Sauer Apr 8 '13 at 15:49
  • 1
    How do you know that just because someone has a 5% reject rate (or 10%, or 50%) that any of their decisions are wrong? If you couldn't skip anything and were forced to always make a decision then yes, the reject ratio should be similar, but since it's not and people can be reviewing very different "pools" of suggested edits it's not even expected (or desirable) that everyone's reject ratio should be similar. – Servy Apr 8 '13 at 15:50
  • @Servy: To think that no one would accept concerns me and it would also concern me if no one would reject. Skewing too much in one direction should dial up the review audits IMHO. – Brian Apr 8 '13 at 15:52
  • 1
    @0A0D Again, you're assuming that people are reviewing everything that passes in front of them. Not only are many people not doing that, they shouldn't be doing that in most cases. It's expected for people to spend time reviewing different types of posts, so they ought to have a certain variance in terms of reject ratio. – Servy Apr 8 '13 at 15:54
  • @Servy: Why not? I question your effectiveness as a reviewer. I would say most of the time, you don't need to be familiar with the content. If you skip alot, then you aren't helping much. – Brian Apr 8 '13 at 16:05
  • @0A0D I agree, you can still categorize reviews. For example, I virtually always skill all tag wiki related suggested edits. Some people might skip edits surrounding very detailed spelling/grammar and focus more on those related to formatting (or the reverse) based on what they feel more knowledgeable about. Some might actively skip more involved edits in favor of those that are easier to determine the correctness of, or do the reverse of that. At the end of the day, so long as all approve/reject votes are correct, you can skip whatever you want. – Servy Apr 8 '13 at 16:08
  • 3
    @0A0D not helping much is fine. We choose how much to help. Actively making the site worse, by making bad review decisions and thus contributing to either rewarding bad edits or punishing good ones, is a Bad Thing. Any gameable target can be gamed - the robo-approver switches from Approve, Approve, Approve, Approve to Approve, Reject, Approve, Approve and barely skips a beat, or waits for a bubble saying "edits need rejecting too" and clicks Reject. No thought, no improvement. This is so much worse than some reviewers choosing not to help much. – Kate Gregory Apr 8 '13 at 16:10
  • @KateGregory: I would think it is hard to outsmart the audit system since the text is non-intuitive and hard for the AI to detect. Ramping up the audits on people who are skewed in either direction would help block the robots who approve, approve, approve because eventually they will be put in the penalty box by failing the additional audits. – Brian Apr 8 '13 at 16:19
  • @0A0D It's not actually that hard for even poor quality reviewers to deal with audits. Most of them are too lazy to do it, but it's not particularly hard. You also need ways of dealing with reviewers who are just bad at reviewing (despite sincerely trying) as opposed to blindly acting without even reading. If the audits were able to remove robo-reviwing in entirety then there would be no need for this suggestion. That this suggestion is here indicates there is still a problem, it's just been mitigated. – Servy Apr 8 '13 at 16:25
  • @Servy: Well, moderators have tools to detect and remove bad reviewers so I don't see why we can't mitigate further. – Brian Apr 8 '13 at 16:32
  • 1
    @0A0D We certainly can mitigate further. My answer is fundamentally saying that this doesn't help discourage bad reviewing, not that nothing needs to be done about the issue. I'm also saying that there is a limit to what can be done through audits; you are assuming audits will solve the problem, we just need more of them. I'm saying that audits have probably done most of what they'll be able to do, and to make the system better we need to do more than just audit. – Servy Apr 8 '13 at 16:38
  • @Servy: I am saying that I want the bans to happen sooner, by forcing the bots or bad reviewers to fail them sooner. – Brian Apr 8 '13 at 16:39
  • @0A0D And I'm saying that the audits are already quite frequent. If the reviewers that have an abnormal accept rate are failing audits then the system will already take care of it and they won't be a problem. The fact that they exist means that they're not failing the audits. The normal audit rate is already quite high; it's not like it's every 50 audits or something really uncommon. – Servy Apr 8 '13 at 16:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .