This does not work. The problem is in the fact that you are choosing to do this based on it happening three times in a row. This assumes that such a bad reviewer does most of his or her reviewer incorrectly.
However, when an edit is approved, the poor reviewer will have reviewed correctly and as such, the counter will be reset according to this plan. As such, you would need three consecutive rejected edits in order to be able to catch anyone in this proposed system.
In my experience the large majority of edits is approved, and as such it will rarely happen that three consecutive edits are rejected. This means this system does not work.
I decided to do some number crunching. I said in the comments I wasn't sure about how to do the math, but at that time I was trying to do some rules I was taught that I both didn't know too well and were hard (if at all possible) to apply to this problem. As it turned out, I just needed to step back and use some general math knowledge instead.
Basically, this involves three cases. First off, you might just have had two rejects last. In this case, you have either the chance
p that it's done by the next edit review (
p * 1) and the inverse chance that we're going to do have an approved edit and have to start over. (
p is the part of the edits that get approved). If we have only just had our first reject in a row, then we have a
p chance we reject another review and from there we go to the two rejects we discussed above, and otherwise we once again have an approved edit and have to start over. Finally if we start without any streak of rejected edits, have a
p chance to advance to the previous case of having a streak of one, and otherwise we do a review and end up right where we started: without a streak.
Now I know that may not have been too clear, but maybe it'll be clearer if I put it in formulas:
M(n): the number of moves we'll expect before three consecutive rejected reviews
when we currently have a running streak of n rejected reviews
M(2): p + (1 - p) * (M(0) + 1)
M(1): p * (M(2) + 1) + (1 - p) * (M(0) + 1)
M(0): p * (M(1) + 1) + (1 - p) * (M(0) + 1)
Substitution gives us:
M(0) = p*(p*(p + (1 - p)*(S0 + 1) + 1) + (1 - p)*(S0 + 1) + 1) + (1 - p)*(S0 + 1)
Which can be rewritten as (I renamed
M here as we're only looking at the whole thing from the beginning now):
M = p^2 + p + 1 + (-p^3 + 1) * M
The first thing I was going to calculate was how long it takes when we only assume that 50% of all edits is rejected. Filling in
p = 0.5 gives you
M = 14, which is exactly what the web has to say about the topic (just google consecutive coin flips). From there, let's add a bit more realism to our model.
Let's say that 45% of edits are unanimously accepted, 45% are unanimously rejected and 10% of all edits are disputed by true reviewers. Let's say that on those question, 50% of all valid reviewers votes to reject and 50% votes to accept. Now let's see what that means for our badge hunter.
A question that our badge hunter sees has him in it. The undisputed questions are simple, but the disputed questions are a little harder. The idea is that whenever he ends up voting to accept a disputed edit, we'll need the first three other reviewers that are reviewing the question to reject it, or otherwise it won't be a 3v1 and won't count against his streak. Like we did silently before, let's assume that there are no other malicious reviewers for now.
The chance of having three consecutive reject votes on a disputed question are
0.5 * 0.5 * 0.5 = 0.125. As such, 87.5% of all disputed questions will be combo breakers. We had 10% disputed questions, so that brings us to 8.75%. So our combo breakers are now
0.45 + 0.0875 = 0.5375, which means that
p drops to
p into our function gives us:
M = 16.945, we're almost down to triggering the system only once a day.
Now let's add other badge hunters into the mix. If we meet another badge hunter on any question that would otherwise be a 3v1, it becomes a combo breaker, as he'll also accept it. Assuming one percent of the reviewers are bad apples, the chance of that happening is:
0.01 * 1 * 1 * 3 = 0.03 As such we have to decrease our 3v1 losses by
0.4625 * 0.03 = 0.013875. Now
p = 0.448725. Fill it in and we get
M = 18.2626.
With the maximum number of reviews per day being 20, this means on average each bad reviewer will get banned only once per day. This means that a single day ban won't do any real good for the system - I believe it may mean a decrease of badge hunter reviews by somewhere between 25% - 50%, but I don't feel like doing the math on that as well (nor do I feel like recombining this result with my previous results and take this into account for the number of other badge hunters encountered). Of course, this system could be manipulated by punishing harder on multiple day-bans in a short period of time, but I think the situation is actually quite a bit worse than what I described here, as the numbers I used are pretty generous.
So can we just up the number of days you are review-banned for? Let's take a look at the false positives for that:
On a disputed question, you have 50% chance to vote to accept. If you do and the first three other people to vote on it are either badge hunters or happen to vote to reject. The chances of this happening on any disputed question are:
0.5 * (0.01 + 0.99 * 0.5) * (0.01 + 0.99 * 0.5) * (0.01 + 0.99 * 0.5) = 0.06439. On a clear reject, the chances of a false positive are
0.01 * 0.01 * 0.01 = 0.000001. The total chance of a false positive on any question our real reviewer reviews is thus
0.06439 * 0.1 + 0.000001 * 0.45 = 0.0064394. Usiong
p = 0.0064394 we get
M = 3769375. A reviewer doing 20 reviews each day would on average be banned for more than a day each
3769375 / 20 = 188468 days. If he does his 20 reviews every single day of the year, that's once every
3769375 / 365 = 10327 years. This sounds acceptable enough to me. Let's take a look at this from one more angle. Going back to
M = 3769375, and let's say this time that a 1000 valid reviews are done every day, giving us an average
3769375 / 1000 = 3769 days before any legitimate reviewer is banned, which still over 10 years, which again is acceptable in my opinion.
So it looks like a ban longer than a single day is needed for enough of a punishment on a three streak of 3v1s. However, one should note that I believe the numbers I used are quite optimistic and as such, I believe you would need a ban of quite a bit more than a day to make this effective. It also looks like the false positive won't be too much of a problem. However, here we have the same problems with the numbers we used and on top of that we're not dealing with fact that some people accept more easily than others, meaning that someone who may not be too good a reviewer but isn't a badge hunter may well be banned by this system. The question is how long a ban you're willing to give this person.
Before we wrap this up, I want to look at one more thing: varying the length of the streak required for a ban. Let's start off with a streak of 2. Here the problem is false positives, so let's look at that. Our function becomes
M = p + 1 (-p^2 + 1) * M Filling in
p = 0.06439 from above, we get
M = 155, meaning that for every eight days a legitimate reviewer does his 20 reviews, he'll get banned once (on average of course). This might be acceptable if you realize that most people spending much time on this website will have a lot of experience so might well have too much experience to be affected by this. However, if we add to the mix that my numbers were optimistic estimates and that some people are more inclined to accept than others without being badge hunters, and I don't think this system holds up. So how about making the streak size 4 then? In that case, our function becomes
M = p^3 + p^2 + p + 1 + (-p^4 + 1) * M. Here the problem is how long it takes to catch our bad guys, so we input
p = 0.448725 from above. We get
M = 42.9275, which is over 2 days and is already getting into dangerous territory. A quick look also show that if the real chance is 0.1 lower, this has the effect of increasing the time to catch a bad reviewer to over 5 days, whereas it would still be about one and a half day with a streak of three. This gets out of hand pretty quickly and I'd say that this is not in any way effective (and we're getting into territory where a ban has to be so long that a single false positive is unacceptable) unless my estimates are actually quite accurate (or the difference is on the other side of what I thought them to be). In brief, I don't think using different streak lengths is a possibility.
In the end, the number crunching provided nothing surprising (to me anyway) but I hope it provides the numbers to back up my original claims that this system doesn't work.
And for good measure, here is all the assumptions I made in my calculations:
45% of all edits are rejected by all good reviewers
45% of all edits are accepted by all good reviewers
10% of all edits are disputed
A good reviewer will accept 50% and reject 50% of all contested questions
(There is no difference between contested questions.)
1% of all reviewers are bad
bad reviewers accept in 100% of cases
Each day, an average of 1000 times someone votes to reject or accept an edit
(For the last one goes that I have only used it in one calculation, which
wasn't too exact anyway)
As per request, here's the math for 25% bad apples:
Once again, we'll add other badge hunters into the mix. The chance of any question that was otherwise going to be a 3v1 having another badge hunter is:
0.25 * 1 * 1 * 3 = 0.75 As such we have to decrease our 3v1 losses by
0.4625 * 0.75 = 0.346875. Now
p = 0.115625. Fill it in and we get
M = 730.359. Basically we won't catch people legitimately.
The chance of a false positive becomes
0.1 * 0.5 * (0.25 + 0.75 * 0.5) * (0.25 + 0.75 * 0.5) * (0.25 + 0.75 * 0.5) + 0.45 * 0.25 * 0.25 * 0.25 = 0.01923 filling it in we get that it takes
143381 edits on average to get a false positive, which is still sort of acceptable until we start adding poor sincere reviewers and the dynamics of the real world.