25

This is basically a -ification of this post of mine

From what I see, Stack Overflow mods are having to spend time manually rooting out robo reviewers. To me, it seems like the system should be managing that pretty well on its own (and given that I still do see bad reviews, this isn't working)

Why not have the frequency of audits vary depending on "suspicion"? Basically, whenever the user fails an audit, the suspicion level increases1. The more suspicious the system is of a user, the more audits the user gets. Passed audits reduce the suspicion level.

This ought to catch robo reviewers more efficiently as even a single failed audit will automatically put them under higher system scrutiny. This won't harm legit reviewers, even if they fail an audit or two, they can recover their status by passing the next few.

1. Other things can add to the suspicion level too, for example when a user makes a review that contradicts the other reviewers, suspicion should increase (slightly, not as much as a failed audit). Another such factor that could be included is posts that are marked as "no action needed" and later on spam deleted.

Improve this question
2
  • 10
    YES
    – Gilles 'SO- stop being evil'
    Commented Sep 7, 2013 at 22:58
  • I like the idea of having qualities other than reputation. Hmmm, I wonder if unaccountably peckish might be needed here too... *8')
    – Mark Booth
    Commented Sep 9, 2013 at 9:38

2 Answers 2

Reset to default
9
+50

This would be a great idea, if the review audit algorithm actually worked.

I posted a very clear example here of why it doesn't, and there was a very similar post made by another user that same day. If the review audits can fail that badly when selecting a question for the audit, it would be too easy for the "suspicion level" to be invalidly increased. I failed my first two review audits ever within a five minute span; the first I questioned (but decided not to follow up on and accept as my mistake), the second was a 100% wrong failure of the audit review itself (and the reason for the post I linked above).

The two review audit failures within 5 minutes certainly would have elevated my "suspicion level" quickly, even though the first IMO was a very questionable failure, and the second was so apparently unreasonable that I asked others to explain it by posting here at Meta. There's no real appeal from an invalid failure, though; I don't think either (or both) of the invalid failures were removed from my history, so I still show that I failed two review audits.

(After an extended discussion in the comments below)

My concern isn't about being banned from reviewing. I review here voluntarily, not because I get anything for doing so, the same way I answer questions.

Having to pass more tests after failing tests that were invalid in the first place is an irritant. My reaction to having failed two invalid audits in a short time span (5 minutes) is "Screw this. I'm not wasting any more time with this (expletive) now." Is that really what we want here?

Improve this answer
40
  • 1
    I got two in five minutes. I highly doubt that I reviewed a dozen posts in that time span, and 2/12 (16.6%) seems like a pretty high percentage to me. And as a very careful reviewer, I find your remark about suspicion and good reviewers somewhat offensive. I assure you I am a quite careful reviewer; as I've stated multiple times, these are the first two review audits I've ever failed, and both were within a five minute span, and one on what apparently several others thought were a very bad audit, as the post was closed in minutes after I posted here about it.
    – Ken White
    Commented Sep 8, 2013 at 4:27
  • 1
    I don't think I agree. I think your proposed system would put me square in the zone of suspicious levels, and there is no way here to appeal invalid review audits. (Yes, I posted here about it and the question was closed, but that doesn't remove the failed audit for me personally, which means I still have two failed audits although one of them was 100% erroneous. When the error rate of review audits is 50%, and there are other complaints about invalid failures in the same time span by other users, any automated "suspicion levels" seems like the SO equivalent of "profiling", and that's wrong.
    – Ken White
    Commented Sep 8, 2013 at 4:42
  • 2
    I might be more agreeable to your suggestion if there were in fact a way to appeal invalid audit failures, but any automated "suspicion level" when there are so many complaints about the validity of audit reviews here just seems wrong. It also seems that mods would be adversely affected, as people would not review as many questions because they could be flagged as "suspicious reviewers" and have no way to appeal. Unless the selection of review questions is improved, or a process of having invalid review failures is in place, any automated "bad guy level" is inappropriate IMO.
    – Ken White
    Commented Sep 8, 2013 at 4:47
  • 2
    ...except that the "suspicion level" doesn't mean much . There's nothing bad about being in the suspicion level; all that means is that you get more audits. And as you're getting audits really fast, it's easy to come out of this suspicion level as, chances are, the new audits will be good audits. Again, the suspicion level increases and decreases, it doesn't just increase and isn't a permanent black mark.
    – Manishearth
    Commented Sep 8, 2013 at 4:59
  • 1
    These levels won't be public, all they do is temporarily increase the number of audits you get. Bad reviewers will fail those audits too and get banned. Good reviewers will pass them (again, because the chances of getting more than 2 bad audits in a row is extremely less), and have their suspicion level reset. While you can appeal a ban due to invalid audits, the system ought to fix itself here -- even if you're in the suspicion zone, the frequency of audits and the extremely low chances of getting even more invalid audits will ensure that you get out of it pretty quickly.
    – Manishearth
    Commented Sep 8, 2013 at 5:00
  • 1
    Basically, sure, you would be put square in the zone of suspicion. However, all this would mean that you would get more audits. And these would most likely be good audits. And passing those would put you back to where you were. And this is before we even consider that the edge cases where someone gets banned due to faulty reviews can be easily appealed on meta.
    – Manishearth
    Commented Sep 8, 2013 at 5:02
  • 3
    "except that you get more audits" is exactly the problem here. Two for two in five minutes is a bad rate for invalid audit failures. Even 50% (1/2) in 5 minutes is a very bad rate. So now I get more audits, which increases the odds of getting another invalid audit question, which increases the suspicion level, which increases the audits... At what point do users say "Screw reviewing questions, because this is really irritating, annoying, and I hate being told I'm wrong when I'm clearly not and I can't have the mistake fixed."?
    – Ken White
    Commented Sep 8, 2013 at 5:07
  • 1
    And to clarify: Both of the audit failures I had were bad. The first one I thought about asking here about, and decided not to do so; after all, it was only a single failure, and it was the only one I'd ever had. OK. I'll let it slide. The next one, so soon after and that I really couldn't make sense of, I questioned here. I really wish now I'd asked about the first one just for the documentary trail it would have left. I don't care that it's not public knowledge; it's invalid information that the mods and system here use adversely in some way, and that's simply wrong.
    – Ken White
    Commented Sep 8, 2013 at 5:12
  • 1
    @KenWhite once again, these are single data points, not something that happens to most people here. How many good audits have you got afterwards?
    – John Dvorak
    Commented Sep 8, 2013 at 5:14
  • 1
    chat.meta.stackoverflow.com/rooms/info/630/…
    – Manishearth
    Commented Sep 8, 2013 at 5:15
  • 1
    @JanDvorak ...and it also increases the rate in which good ones appear. All that matters is how many you fail in a row, so this doesn't change anything at all. That rate would get reset in a jiffy.
    – Manishearth
    Commented Sep 8, 2013 at 5:18
  • 2
    You're entirely missing my point. I could care less about the possibility of being banned from reviewing. I review voluntarily here, not to get anything out of it, the same way I answer questions. Increasing the annoyance level discourages reviews, and having it happen automatically based on failures caused by invalid audits simply increases the annoyance level.
    – Ken White
    Commented Sep 8, 2013 at 5:20
  • 1
    :-) I think we're butting heads here. The point I'm trying to make is that having to pass more tests after failing tests that were invalid in the first place is an irritant. My reaction to having failed two invalid audits in a short time span is "Screw this. I'm not wasting any more time with this (expletive) now." Is that really what we want here?
    – Ken White
    Commented Sep 8, 2013 at 5:24
  • 1
    FWIW this proposal also tries to reduce the number of audits "good" reviewers get as well, so if you've passed other audits or are reviewing nicely then you may get very few audits anyway.
    – Manishearth
    Commented Sep 8, 2013 at 5:26
  • 2
    @Manishearth: I'll edit to include the irritant factor in my answer. :-)
    – Ken White
    Commented Sep 8, 2013 at 5:29
3

So, I failed a review audit once. It was months ago.

Since then I've reviewed probably hundreds more and I'm sure answered at least a couple dozen review audits. (I also reject about 45% of suggested edits, which if anything is a little high.)

It seems like every 10–20 reviews I encounter a review audit, and since I always pass them, it seems like a waste of effort.

What about tweaking this on the other end? When the last n audits have passed, can you reduce the frequency of audits, say, to 1 in 100?

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .