Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 153 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

This is basically a -ification of this post of mine

From what I see, Stack Overflow mods are having to spend time manually rooting out robo reviewers. To me, it seems like the system should be managing that pretty well on its own (and given that I still do see bad reviews, this isn't working)

Why not have the frequency of audits vary depending on "suspicion"? Basically, whenever the user fails an audit, the suspicion level increases1. The more suspicious the system is of a user, the more audits the user gets. Passed audits reduce the suspicion level.

This ought to catch robo reviewers more efficiently as even a single failed audit will automatically put them under higher system scrutiny. This won't harm legit reviewers, even if they fail an audit or two, they can recover their status by passing the next few.

1. Other things can add to the suspicion level too, for example when a user makes a review that contradicts the other reviewers, suspicion should increase (slightly, not as much as a failed audit). Another such factor that could be included is posts that are marked as "no action needed" and later on spam deleted.

share|improve this question
YES – Gilles Sep 7 '13 at 22:58
I like the idea of having qualities other than reputation. Hmmm, I wonder if unaccountably peckish might be needed here too... *8') – Mark Booth Sep 9 '13 at 9:38

This would be a great idea, if the review audit algorithm actually worked.

I posted a very clear example here of why it doesn't, and there was a very similar post made by another user that same day. If the review audits can fail that badly when selecting a question for the audit, it would be too easy for the "suspicion level" to be invalidly increased. I failed my first two review audits ever within a five minute span; the first I questioned (but decided not to follow up on and accept as my mistake), the second was a 100% wrong failure of the audit review itself (and the reason for the post I linked above).

The two review audit failures within 5 minutes certainly would have elevated my "suspicion level" quickly, even though the first IMO was a very questionable failure, and the second was so apparently unreasonable that I asked others to explain it by posting here at Meta. There's no real appeal from an invalid failure, though; I don't think either (or both) of the invalid failures were removed from my history, so I still show that I failed two review audits.

(After an extended discussion in the comments below)

My concern isn't about being banned from reviewing. I review here voluntarily, not because I get anything for doing so, the same way I answer questions.

Having to pass more tests after failing tests that were invalid in the first place is an irritant. My reaction to having failed two invalid audits in a short time span (5 minutes) is "Screw this. I'm not wasting any more time with this (expletive) now." Is that really what we want here?

share|improve this answer
I got two in five minutes. I highly doubt that I reviewed a dozen posts in that time span, and 2/12 (16.6%) seems like a pretty high percentage to me. And as a very careful reviewer, I find your remark about suspicion and good reviewers somewhat offensive. I assure you I am a quite careful reviewer; as I've stated multiple times, these are the first two review audits I've ever failed, and both were within a five minute span, and one on what apparently several others thought were a very bad audit, as the post was closed in minutes after I posted here about it. – Ken White Sep 8 '13 at 4:27
I don't think I agree. I think your proposed system would put me square in the zone of suspicious levels, and there is no way here to appeal invalid review audits. (Yes, I posted here about it and the question was closed, but that doesn't remove the failed audit for me personally, which means I still have two failed audits although one of them was 100% erroneous. When the error rate of review audits is 50%, and there are other complaints about invalid failures in the same time span by other users, any automated "suspicion levels" seems like the SO equivalent of "profiling", and that's wrong. – Ken White Sep 8 '13 at 4:42
I might be more agreeable to your suggestion if there were in fact a way to appeal invalid audit failures, but any automated "suspicion level" when there are so many complaints about the validity of audit reviews here just seems wrong. It also seems that mods would be adversely affected, as people would not review as many questions because they could be flagged as "suspicious reviewers" and have no way to appeal. Unless the selection of review questions is improved, or a process of having invalid review failures is in place, any automated "bad guy level" is inappropriate IMO. – Ken White Sep 8 '13 at 4:47
...except that the "suspicion level" doesn't mean much . There's nothing bad about being in the suspicion level; all that means is that you get more audits. And as you're getting audits really fast, it's easy to come out of this suspicion level as, chances are, the new audits will be good audits. Again, the suspicion level increases and decreases, it doesn't just increase and isn't a permanent black mark. – Manishearth Sep 8 '13 at 4:59
These levels won't be public, all they do is temporarily increase the number of audits you get. Bad reviewers will fail those audits too and get banned. Good reviewers will pass them (again, because the chances of getting more than 2 bad audits in a row is extremely less), and have their suspicion level reset. While you can appeal a ban due to invalid audits, the system ought to fix itself here -- even if you're in the suspicion zone, the frequency of audits and the extremely low chances of getting even more invalid audits will ensure that you get out of it pretty quickly. – Manishearth Sep 8 '13 at 5:00
Basically, sure, you would be put square in the zone of suspicion. However, all this would mean that you would get more audits. And these would most likely be good audits. And passing those would put you back to where you were. And this is before we even consider that the edge cases where someone gets banned due to faulty reviews can be easily appealed on meta. – Manishearth Sep 8 '13 at 5:02
"except that you get more audits" is exactly the problem here. Two for two in five minutes is a bad rate for invalid audit failures. Even 50% (1/2) in 5 minutes is a very bad rate. So now I get more audits, which increases the odds of getting another invalid audit question, which increases the suspicion level, which increases the audits... At what point do users say "Screw reviewing questions, because this is really irritating, annoying, and I hate being told I'm wrong when I'm clearly not and I can't have the mistake fixed."? – Ken White Sep 8 '13 at 5:07
And to clarify: Both of the audit failures I had were bad. The first one I thought about asking here about, and decided not to do so; after all, it was only a single failure, and it was the only one I'd ever had. OK. I'll let it slide. The next one, so soon after and that I really couldn't make sense of, I questioned here. I really wish now I'd asked about the first one just for the documentary trail it would have left. I don't care that it's not public knowledge; it's invalid information that the mods and system here use adversely in some way, and that's simply wrong. – Ken White Sep 8 '13 at 5:12
@KenWhite once again, these are single data points, not something that happens to most people here. How many good audits have you got afterwards? – Jan Dvorak Sep 8 '13 at 5:14
@JanDvorak ...and it also increases the rate in which good ones appear. All that matters is how many you fail in a row, so this doesn't change anything at all. That rate would get reset in a jiffy. – Manishearth Sep 8 '13 at 5:18
You're entirely missing my point. I could care less about the possibility of being banned from reviewing. I review voluntarily here, not to get anything out of it, the same way I answer questions. Increasing the annoyance level discourages reviews, and having it happen automatically based on failures caused by invalid audits simply increases the annoyance level. – Ken White Sep 8 '13 at 5:20
:-) I think we're butting heads here. The point I'm trying to make is that having to pass more tests after failing tests that were invalid in the first place is an irritant. My reaction to having failed two invalid audits in a short time span is "Screw this. I'm not wasting any more time with this (expletive) now." Is that really what we want here? – Ken White Sep 8 '13 at 5:24
FWIW this proposal also tries to reduce the number of audits "good" reviewers get as well, so if you've passed other audits or are reviewing nicely then you may get very few audits anyway. – Manishearth Sep 8 '13 at 5:26
@Manishearth: I'll edit to include the irritant factor in my answer. :-) – Ken White Sep 8 '13 at 5:29

So, I failed a review audit once. It was months ago.

Since then I've reviewed probably hundreds more and I'm sure answered at least a couple dozen review audits. (I also reject about 45% of suggested edits, which if anything is a little high.)

It seems like every 10–20 reviews I encounter a review audit, and since I always pass them, it seems like a waste of effort.

What about tweaking this on the other end? When the last n audits have passed, can you reduce the frequency of audits, say, to 1 in 100?

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .