Dealing with misguided reviewers of suggested edits - take 2

Question

It was already discussed before with some suggestions (even from me) but here is a final and simple suggestion that in my opinion will hunt down the badge hunters.

Now that the number of reviewers was raised to at least 3 on Stack Overflow, when there is a suggested edit rejected by three and approved by one user we can assume for almost certain that the one user was utterly wrong; either badge hunting, random vote just for fun or whatever.

I suggest that same way that users are blocked from suggesting further edits for X days after Y suggestions are rejected, users who were in 1-3 minority in the review for A times in a row, will be blocked from reviewing any further suggested edits for B hours, instead getting a friendly message.

Sensible numbers are 3 for A and 24 for B, but it doesn't really matter as long as something is done with those users.

Can't do without example so here they are. Same user, several really bad approvals. Exhibit #1 and Exhibit #2. Don't think I need to explain why those two edits are invalid.

Note: unlike other past suggestions, I'm not asking to check approve/reject rate of the user; it's totally fine to only approve suggestions without ever rejecting, as long as all the approvals are valid.

Answer 1

This is all a probability game, right? Hypothetically, if 1% of reviewers are bad, then the chance of this working would be

99% * 99% * 1% * 3 = 2.94% (two correct reviews and one incorrect review)

while the chance of a false positive would be

1% * 1% * 99% * 3 = 0.03% (two incorrect reviewers overruling a correct reviewer)

The rest of the time, you get agreement, in which case the review doesn't count against anyone. Okay, not super-effective, but it would probably help some in the long run. If you increase the number of bad reviewers to 20%, the numbers become

80% * 80% * 80% = 51.2% (good result: correct review)
80% * 80% * 20% * 3 = 38.4% (best result: correct review and bad reviewer caught)
20% * 20% * 80% * 3 = 9.6% (worst result: wrong review and good reviewer caught)
20% * 20% * 20% = 0.8% (bad result: incorrect review)

How often do we get bad reviews in practice? Remember, the figure has to include not only idiot badge grinders but also well-meaning users who just aren't good at reviewing and occasional bad choices from good reviewers. Twenty percent doesn't seem unreasonable, and at that point, this feature would ding a good reviewer once for every four times it gets a bad one.

Answer 2

So what is the underlying problem here, given that we agree that there are poor quality reviewers in the queue frequently performing the incorrect review action?

Is the problem that posts are being approved when they should be rejected, or rejected when they should be approved? That's what I would assert is the main problem.

Another problem that some people seem to have is that people are getting shiny badges when they didn't actually do the work the badge is designed to recognize. Personally, I don't much care about this. I'd be willing to give everyone the gold reviewer badge for free if it meant perfect reviews (unfortunately it won't).

If we build a system by which we assume that the majority is correct, and therefore punish those not voting with the majority, then we're saying that all, or almost all, of the items going through the current review queue are being properly handled. Some people, here or there, are performing the wrong action, but it's never enough to actually result in the majority action being incorrect. That is the only situation in which it would be appropriate to punish the minority voter.

If that's the situation we're in then we don't even have a problem. The correct actions are being taken, so there is nothing to worry about. A few people might get an undeserved badge here or there, but the review process is still solving it's actual duty of improving the content on the site.

If we still have a problem then the problem is a result of there being enough poor reviewers that they actually make a majority decision to perform the wrong action, and they do it frequently enough to make a significant difference. This means that if we have a problem at all that this proposed solution won't actually help solve it. It would mean that "correct" reviewers would be punished and incorrect reviewers would not.

So if there is a problem it doesn't help solve it (or even makes it worse) and if there is no problem then well...there's no problem and we don't need to add a solution.

There is no case where implementing this proposed change would result in improving the site.

Answer 3

This does not work. The problem is in the fact that you are choosing to do this based on it happening three times in a row. This assumes that such a bad reviewer does most of his or her reviewer incorrectly.

However, when an edit is approved, the poor reviewer will have reviewed correctly and as such, the counter will be reset according to this plan. As such, you would need three consecutive rejected edits in order to be able to catch anyone in this proposed system.

In my experience the large majority of edits is approved, and as such it will rarely happen that three consecutive edits are rejected. This means this system does not work.

---

I decided to do some number crunching. I said in the comments I wasn't sure about how to do the math, but at that time I was trying to do some rules I was taught that I both didn't know too well and were hard (if at all possible) to apply to this problem. As it turned out, I just needed to step back and use some general math knowledge instead.

Basically, this involves three cases. First off, you might just have had two rejects last. In this case, you have either the chance p that it's done by the next edit review (p * 1) and the inverse chance that we're going to do have an approved edit and have to start over. (p is the part of the edits that get approved). If we have only just had our first reject in a row, then we have a p chance we reject another review and from there we go to the two rejects we discussed above, and otherwise we once again have an approved edit and have to start over. Finally if we start without any streak of rejected edits, have a p chance to advance to the previous case of having a streak of one, and otherwise we do a review and end up right where we started: without a streak.

Now I know that may not have been too clear, but maybe it'll be clearer if I put it in formulas:

M(n): the number of moves we'll expect before three consecutive rejected reviews
      when we currently have a running streak of n rejected reviews
M(2): p + (1 - p) * (M(0) + 1)
M(1): p * (M(2) + 1) + (1 - p) * (M(0) + 1)
M(0): p * (M(1) + 1) + (1 - p) * (M(0) + 1)

Substitution gives us:

M(0) = p*(p*(p + (1 - p)*(S0 + 1) + 1) + (1 - p)*(S0 + 1) + 1) + (1 - p)*(S0 + 1)

Which can be rewritten as (I renamed M(0) to M here as we're only looking at the whole thing from the beginning now):

M = p^2 + p + 1 + (-p^3 + 1) * M

The first thing I was going to calculate was how long it takes when we only assume that 50% of all edits is rejected. Filling in p = 0.5 gives you M = 14, which is exactly what the web has to say about the topic (just google consecutive coin flips). From there, let's add a bit more realism to our model.

Let's say that 45% of edits are unanimously accepted, 45% are unanimously rejected and 10% of all edits are disputed by true reviewers. Let's say that on those question, 50% of all valid reviewers votes to reject and 50% votes to accept. Now let's see what that means for our badge hunter.

A question that our badge hunter sees has him in it. The undisputed questions are simple, but the disputed questions are a little harder. The idea is that whenever he ends up voting to accept a disputed edit, we'll need the first three other reviewers that are reviewing the question to reject it, or otherwise it won't be a 3v1 and won't count against his streak. Like we did silently before, let's assume that there are no other malicious reviewers for now.

The chance of having three consecutive reject votes on a disputed question are 0.5 * 0.5 * 0.5 = 0.125. As such, 87.5% of all disputed questions will be combo breakers. We had 10% disputed questions, so that brings us to 8.75%. So our combo breakers are now 0.45 + 0.0875 = 0.5375, which means that p drops to 0.4625. Throwing p into our function gives us: M = 16.945, we're almost down to triggering the system only once a day.

Now let's add other badge hunters into the mix. If we meet another badge hunter on any question that would otherwise be a 3v1, it becomes a combo breaker, as he'll also accept it. Assuming one percent of the reviewers are bad apples, the chance of that happening is: 0.01 * 1 * 1 * 3 = 0.03 As such we have to decrease our 3v1 losses by 0.4625 * 0.03 = 0.013875. Now p = 0.448725. Fill it in and we get M = 18.2626.

With the maximum number of reviews per day being 20, this means on average each bad reviewer will get banned only once per day. This means that a single day ban won't do any real good for the system - I believe it may mean a decrease of badge hunter reviews by somewhere between 25% - 50%, but I don't feel like doing the math on that as well (nor do I feel like recombining this result with my previous results and take this into account for the number of other badge hunters encountered). Of course, this system could be manipulated by punishing harder on multiple day-bans in a short period of time, but I think the situation is actually quite a bit worse than what I described here, as the numbers I used are pretty generous.

So can we just up the number of days you are review-banned for? Let's take a look at the false positives for that:

On a disputed question, you have 50% chance to vote to accept. If you do and the first three other people to vote on it are either badge hunters or happen to vote to reject. The chances of this happening on any disputed question are: 0.5 * (0.01 + 0.99 * 0.5) * (0.01 + 0.99 * 0.5) * (0.01 + 0.99 * 0.5) = 0.06439. On a clear reject, the chances of a false positive are 0.01 * 0.01 * 0.01 = 0.000001. The total chance of a false positive on any question our real reviewer reviews is thus 0.06439 * 0.1 + 0.000001 * 0.45 = 0.0064394. Usiong p = 0.0064394 we get M = 3769375. A reviewer doing 20 reviews each day would on average be banned for more than a day each 3769375 / 20 = 188468 days. If he does his 20 reviews every single day of the year, that's once every 3769375 / 365 = 10327 years. This sounds acceptable enough to me. Let's take a look at this from one more angle. Going back to M = 3769375, and let's say this time that a 1000 valid reviews are done every day, giving us an average 3769375 / 1000 = 3769 days before any legitimate reviewer is banned, which still over 10 years, which again is acceptable in my opinion.

So it looks like a ban longer than a single day is needed for enough of a punishment on a three streak of 3v1s. However, one should note that I believe the numbers I used are quite optimistic and as such, I believe you would need a ban of quite a bit more than a day to make this effective. It also looks like the false positive won't be too much of a problem. However, here we have the same problems with the numbers we used and on top of that we're not dealing with fact that some people accept more easily than others, meaning that someone who may not be too good a reviewer but isn't a badge hunter may well be banned by this system. The question is how long a ban you're willing to give this person.

Before we wrap this up, I want to look at one more thing: varying the length of the streak required for a ban. Let's start off with a streak of 2. Here the problem is false positives, so let's look at that. Our function becomes M = p + 1 (-p^2 + 1) * M Filling in p = 0.06439 from above, we get M = 155, meaning that for every eight days a legitimate reviewer does his 20 reviews, he'll get banned once (on average of course). This might be acceptable if you realize that most people spending much time on this website will have a lot of experience so might well have too much experience to be affected by this. However, if we add to the mix that my numbers were optimistic estimates and that some people are more inclined to accept than others without being badge hunters, and I don't think this system holds up. So how about making the streak size 4 then? In that case, our function becomes M = p^3 + p^2 + p + 1 + (-p^4 + 1) * M. Here the problem is how long it takes to catch our bad guys, so we input p = 0.448725 from above. We get M = 42.9275, which is over 2 days and is already getting into dangerous territory. A quick look also show that if the real chance is 0.1 lower, this has the effect of increasing the time to catch a bad reviewer to over 5 days, whereas it would still be about one and a half day with a streak of three. This gets out of hand pretty quickly and I'd say that this is not in any way effective (and we're getting into territory where a ban has to be so long that a single false positive is unacceptable) unless my estimates are actually quite accurate (or the difference is on the other side of what I thought them to be). In brief, I don't think using different streak lengths is a possibility.

In the end, the number crunching provided nothing surprising (to me anyway) but I hope it provides the numbers to back up my original claims that this system doesn't work.

And for good measure, here is all the assumptions I made in my calculations:

45% of all edits are rejected by all good reviewers
45% of all edits are accepted by all good reviewers
10% of all edits are disputed
A good reviewer will accept 50% and reject 50% of all contested questions
(There is no difference between contested questions.)
1% of all reviewers are bad
bad reviewers accept in 100% of cases
Each day, an average of 1000 times someone votes to reject or accept an edit
(For the last one goes that I have only used it in one calculation, which
 wasn't too exact anyway)

---

As per request, here's the math for 25% bad apples:

Once again, we'll add other badge hunters into the mix. The chance of any question that was otherwise going to be a 3v1 having another badge hunter is: 0.25 * 1 * 1 * 3 = 0.75 As such we have to decrease our 3v1 losses by 0.4625 * 0.75 = 0.346875. Now p = 0.115625. Fill it in and we get M = 730.359. Basically we won't catch people legitimately.

The chance of a false positive becomes 0.1 * 0.5 * (0.25 + 0.75 * 0.5) * (0.25 + 0.75 * 0.5) * (0.25 + 0.75 * 0.5) + 0.45 * 0.25 * 0.25 * 0.25 = 0.01923 filling it in we get that it takes 143381 edits on average to get a false positive, which is still sort of acceptable until we start adding poor sincere reviewers and the dynamics of the real world.

Answer 4

I do think a quality review system based on statistics is appropriate, but using the pure number of disagreements with the crowd rather than relative number of times they disagree is a bad call. Especially when you consider that out of the two examples you hand-selected to demonstrate the need to punish minority reviewers, one of them is wrong! #Exhibit 1 is a perfectly legitimate edit that should have been accepted! The reviewer removed extraneous code that wasn't being referenced anywhere and therefore wasn't germane to the question in a generalized sense. We WANT people to do this type of cleanup, but three reviewers looked at it, didn't bother to understand why lines of code were being removed and said "Nope!"... it's the majority who are wrong here, and this happens A LOT.

With some non-trivial frequency, the obvious review is not the correct review, so we need to be very very careful with any filter that blindly presumes that the popular vote is the correct vote.

EDIT

I want to clarify that just because our friend TheTXI was correct and the majority was wrong in Exhibit 1, it does not make TheTXI a good reviewer. His approval on Exhibit 2 is basically unforgivable: Approving an incorrect edit to the code of a four-year old accepted answer to a core question with 95 upvotes, and which is likely a core resource for many programmers is downright dangerous. I suspect that his reasonable approval in case #1 was incidental rather than intentional and there should be a system for hitting his rep or suspending his review privileges. It just shouldn't rely on a simple and often-wrong heuristic like disagreeing with the majority. There's a huge amount of subjectivity in some of the review process (definitions of "trivial" vary by reviewer; some people prefer that certain types of edits be referred back to the original author as a comment rather than making the correction themselves; the term "off topic" has as many definitions as there are reviewers, etc etc).

We clearly need to do something to improve the review process, but basing it on whether you agree with the majority is not it.

Answer 5

In the name of fun and games, why not have the minority reviewer(s) "donate" rep to the other reviewers? This will "encourage" reviewers to carefully consider the actions of their peers and affirm the majority decision.