Timeline for Vary the number of audits depending on the review history of a user
Current License: CC BY-SA 3.0
47 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 20, 2017 at 10:30 | history | edited | CommunityBot |
replaced http://meta.stackexchange.com/ with https://meta.stackexchange.com/
|
|
| Apr 24, 2014 at 13:50 | history | edited | CommunityBot |
Migration of MSO links to MSE links
|
|
| Jan 14, 2014 at 11:57 | history | bounty ended | Amicable | ||
| Jan 9, 2014 at 5:31 | comment | added | jmac | Way late to the party, but I have to agree with @Manis here. Unless over 50% of review audits in any given queue are bad, careful reviewing will make you less likely to get audits in the future, and decrease the irritating factor. Let's say 20% are bad. After doing 10 audits, it would be virtually impossible to have a negative score if you are reviewing the audits carefully. | |
| Sep 8, 2013 at 18:59 | comment | added | Manishearth | Anyway, the only real disagreement between us seems to be the frequency of the invalid audits. There's ... not much of a point arguing on that anymore, as neither of us have anything to back it up :) | |
| Sep 8, 2013 at 18:58 | comment | added | Manishearth | @KenWhite The suspicion level is based on actions made by the user on an action that the system took. And again, invalid reviews are not as common as you seem to portray, so having more than two in a row is extremely rare. (Two in a row itself is a very rare edge case imo, though it happened to you). | |
| Sep 8, 2013 at 18:55 | comment | added | Ken White | @Manishearth: Sorry; my last comment was intended to respond to your comment about profiling just above, but I failed to indicate so. | |
| Sep 8, 2013 at 14:51 | comment | added | Ken White | Yes,it is. Automated flag bans are based on actions the user made, as are consecutive closed votes. (We users closed the posts, and the system detected the number and timing we did so.) The suspicion level is not based on the same thing - it's based wholly on actions the system took, including invalid testing material that produced the problem in the first place.) "Here's a game designed so you can't win, and losing results in punishment. While we're ay it, we'll annoy you and frustrate you, and at times just make you really ticked off. Wanna play? Better yet, wanna play again later too?" | |
| Sep 8, 2013 at 7:39 | comment | added | Manishearth | @KenWhite Re:generic profiling: But the audit system itself is profiling then. So is the question ban system. So are the automated flags for consecutive closed posts. I agree with the annoyance bit, but I don't see why this is "profiling". Especially when it's temporary. | |
| Sep 8, 2013 at 7:06 | history | edited | Ken White | CC BY-SA 3.0 |
Removed some extraneous information added in my last edit
|
| Sep 8, 2013 at 5:43 | comment | added | Ken White | @Jan: I'd still object (although maybe not as strenuously, because the levels in the other queues is so low I don't need to spend much time there <g>). Any sort of automated identification of problems based on what to me (and based on other posts here many others) is a flawed audit system just seems wrong, in the same way other sorts of generic profiling for behavior seems wrong. | |
| Sep 8, 2013 at 5:37 | history | edited | Ken White | CC BY-SA 3.0 |
Clarified the reason for my objection to this change
|
| Sep 8, 2013 at 5:29 | comment | added | Ken White | @Manishearth: I'll edit to include the irritant factor in my answer. :-) | |
| Sep 8, 2013 at 5:29 | comment | added | John Dvorak | @KenWhite what if this is enabled only outside of the close queue? | |
| Sep 8, 2013 at 5:28 | comment | added | Ken White | @Jan: The close review queue is part of what we're discussing. See the post I linked in my reply above. As the close queue is the one with 70K+ items in it, and consistently stays high, it's an important part of the discussion. Invalid close votes can be reversed very quickly by other users. Do we really want to discourage people from helping to clean up that queue? | |
| Sep 8, 2013 at 5:26 | comment | added | John Dvorak | meaning fewer bad audits for good reviewers | |
| Sep 8, 2013 at 5:26 | comment | added | Manishearth | FWIW this proposal also tries to reduce the number of audits "good" reviewers get as well, so if you've passed other audits or are reviewing nicely then you may get very few audits anyway. | |
| Sep 8, 2013 at 5:24 | comment | added | Manishearth | Ah, I see. Could you make that more explicit in your answer? That's a valid concern, getting audits over a shorter period of time is more annoying than getting them over a longer timespan. I still feel that what you experienced is a major edge case (and won't happen to others), but that's just me. | |
| Sep 8, 2013 at 5:24 | comment | added | Ken White | :-) I think we're butting heads here. The point I'm trying to make is that having to pass more tests after failing tests that were invalid in the first place is an irritant. My reaction to having failed two invalid audits in a short time span is "Screw this. I'm not wasting any more time with this (expletive) now." Is that really what we want here? | |
| Sep 8, 2013 at 5:24 | comment | added | John Dvorak | @KenWhite the close vote review queue is about the only place where we need to actively encourage more reviews. LQ/LA/RO are always empty, and the edit queue doesn't have bad audits. I wouldn't worry about annoyance levels. | |
| Sep 8, 2013 at 5:20 | comment | added | Ken White | You're entirely missing my point. I could care less about the possibility of being banned from reviewing. I review voluntarily here, not to get anything out of it, the same way I answer questions. Increasing the annoyance level discourages reviews, and having it happen automatically based on failures caused by invalid audits simply increases the annoyance level. | |
| Sep 8, 2013 at 5:18 | comment | added | Manishearth | @JanDvorak ...and it also increases the rate in which good ones appear. All that matters is how many you fail in a row, so this doesn't change anything at all. That rate would get reset in a jiffy. | |
| Sep 8, 2013 at 5:15 | comment | added | John Dvorak | @Manishearth a proposal that increases the audit rate does not change the probability of a specific audit being a bad one, but it increases the rate at which the bad ones appear. | |
| Sep 8, 2013 at 5:15 | comment | added | Manishearth | chat.meta.stackoverflow.com/rooms/info/630/… | |
| Sep 8, 2013 at 5:14 | comment | added | John Dvorak | @KenWhite once again, these are single data points, not something that happens to most people here. How many good audits have you got afterwards? | |
| Sep 8, 2013 at 5:14 | comment | added | Manishearth | I understand, and I sympathize with you. The point I'm trying to make here is that getting enough invalid audits in a row to get a ban is a very rare event. And that my proposal does not change the probability of getting X invalid audits in a row | |
| Sep 8, 2013 at 5:13 | comment | added | Manishearth | Getting more audits does not exactly increase the odds of getting an invalid audit question, it increases the odds of getting an audit period (which also increases the odds of getting a good audit question -- and by a greater factor). And passing audits gets the user out of the rut. | |
| Sep 8, 2013 at 5:12 | comment | added | Ken White | And to clarify: Both of the audit failures I had were bad. The first one I thought about asking here about, and decided not to do so; after all, it was only a single failure, and it was the only one I'd ever had. OK. I'll let it slide. The next one, so soon after and that I really couldn't make sense of, I questioned here. I really wish now I'd asked about the first one just for the documentary trail it would have left. I don't care that it's not public knowledge; it's invalid information that the mods and system here use adversely in some way, and that's simply wrong. | |
| Sep 8, 2013 at 5:12 | comment | added | John Dvorak | OK... where's the "convert to a chat room" link? | |
| Sep 8, 2013 at 5:11 | comment | added | Manishearth | I repeat: "Except that you get more audits" won't last long for good reviewers, the more audits will get passed, because invalid audits are rare. 2 out of 12 is a very improbable case. | |
| Sep 8, 2013 at 5:10 | comment | added | Manishearth | Again, two for two in five minutes is not the norm. And this doesn't increase one's chances of getting eventually banned at all. This just makes a user get banned faster. Let's say a (good) user failed two invalid audits. In the current system, if the next audit was invalid and the user failed it, that would probably trigger a ban. Just that the "next audit" may take a while to appear. In the proposed system, the "next audit" will turn up pretty quickly. The chances of getting review banned are the same, the speed is more. | |
| Sep 8, 2013 at 5:07 | comment | added | Manishearth | @JanDvorak I mean "in the other queues". Yes, the suggested edit queue audits are quite well-crafted, so this would work well there. However, I feel that it should be in all queues which have robo reviewer problems (i.e. all of them, currently, though the close vote queue is better off than most of the others.). | |
| Sep 8, 2013 at 5:07 | comment | added | Ken White | "except that you get more audits" is exactly the problem here. Two for two in five minutes is a bad rate for invalid audit failures. Even 50% (1/2) in 5 minutes is a very bad rate. So now I get more audits, which increases the odds of getting another invalid audit question, which increases the suspicion level, which increases the audits... At what point do users say "Screw reviewing questions, because this is really irritating, annoying, and I hate being told I'm wrong when I'm clearly not and I can't have the mistake fixed."? | |
| Sep 8, 2013 at 5:04 | comment | added | John Dvorak | @Manishearth I have yet to see an edit audit worth accepting (and you won't think "hey, I can reverse that manually, then edit the correct stuff in" twice). I have done more than 2.7k edits. All of the audits I got were pure vandalism. | |
| Sep 8, 2013 at 5:02 | comment | added | Manishearth | @JanDvorak Hmm, dunno. I still see bad reviews all over the place, but minitech would know more about this. | |
| Sep 8, 2013 at 5:02 | comment | added | Manishearth | Basically, sure, you would be put square in the zone of suspicion. However, all this would mean that you would get more audits. And these would most likely be good audits. And passing those would put you back to where you were. And this is before we even consider that the edge cases where someone gets banned due to faulty reviews can be easily appealed on meta. | |
| Sep 8, 2013 at 5:02 | comment | added | John Dvorak | The audit system works perfectly in the suggested edit queue - and bad edits are a critical thing to prevent. What if this is enabled only there? | |
| Sep 8, 2013 at 5:00 | comment | added | Manishearth | These levels won't be public, all they do is temporarily increase the number of audits you get. Bad reviewers will fail those audits too and get banned. Good reviewers will pass them (again, because the chances of getting more than 2 bad audits in a row is extremely less), and have their suspicion level reset. While you can appeal a ban due to invalid audits, the system ought to fix itself here -- even if you're in the suspicion zone, the frequency of audits and the extremely low chances of getting even more invalid audits will ensure that you get out of it pretty quickly. | |
| Sep 8, 2013 at 4:59 | comment | added | Manishearth | ...except that the "suspicion level" doesn't mean much . There's nothing bad about being in the suspicion level; all that means is that you get more audits. And as you're getting audits really fast, it's easy to come out of this suspicion level as, chances are, the new audits will be good audits. Again, the suspicion level increases and decreases, it doesn't just increase and isn't a permanent black mark. | |
| Sep 8, 2013 at 4:47 | comment | added | Ken White | I might be more agreeable to your suggestion if there were in fact a way to appeal invalid audit failures, but any automated "suspicion level" when there are so many complaints about the validity of audit reviews here just seems wrong. It also seems that mods would be adversely affected, as people would not review as many questions because they could be flagged as "suspicious reviewers" and have no way to appeal. Unless the selection of review questions is improved, or a process of having invalid review failures is in place, any automated "bad guy level" is inappropriate IMO. | |
| Sep 8, 2013 at 4:42 | comment | added | Ken White | I don't think I agree. I think your proposed system would put me square in the zone of suspicious levels, and there is no way here to appeal invalid review audits. (Yes, I posted here about it and the question was closed, but that doesn't remove the failed audit for me personally, which means I still have two failed audits although one of them was 100% erroneous. When the error rate of review audits is 50%, and there are other complaints about invalid failures in the same time span by other users, any automated "suspicion levels" seems like the SO equivalent of "profiling", and that's wrong. | |
| Sep 8, 2013 at 4:35 | comment | added | Manishearth | I'm not insinuating that you're a bad reviewer, just saying that the two-faulty-audit thing would not have affected you adversely in the proposed system. | |
| Sep 8, 2013 at 4:34 | comment | added | Manishearth | That's in your case. I'm talking about overall -- to me getting 2 bad audits out of 12 normal posts itself is an extremely rare situation. Being a large site, that rare case had to happen to someone, and that was you. I'm not trying to offend here, I'm saying that if you fell into the suspicion zone due to faulty audits, there is an even lesser chance that you would be shown faulty audits afterwards so it should be easy to get out of the zone. Getting 4 faulty audits in a row seems extremely rare to me. | |
| Sep 8, 2013 at 4:27 | comment | added | Ken White | I got two in five minutes. I highly doubt that I reviewed a dozen posts in that time span, and 2/12 (16.6%) seems like a pretty high percentage to me. And as a very careful reviewer, I find your remark about suspicion and good reviewers somewhat offensive. I assure you I am a quite careful reviewer; as I've stated multiple times, these are the first two review audits I've ever failed, and both were within a five minute span, and one on what apparently several others thought were a very bad audit, as the post was closed in minutes after I posted here about it. | |
| Sep 8, 2013 at 4:16 | comment | added | Manishearth | I do agree that there are many faulty audit pages, but they are less in number. Remember, all this is doing is increasing the number of audits you get until you pass a couple again. I think it's pretty unlikely that after getting one or two bad audits, the next few audits (which will rapidly turn up) will be bad as well. A good reviewer ought to be able to escape the high-suspicion zone with ease. (Besides, if suspicion works the other way as well -- passed audits lower suspicion -- then a good reviewer doesn't need to worry about audits at all) | |
| Sep 8, 2013 at 3:56 | history | edited | Ken White | CC BY-SA 3.0 |
Minor clarification about first failure (in first paragraph)
|
| Sep 7, 2013 at 23:08 | history | answered | Ken White | CC BY-SA 3.0 |