What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

Today I came across an OBVIOUSLY good answer in /review. It had 7 upvotes, contained lists, paragraphs of explanation, and well-applied sample code. It was also moderately long. How does a question like this get flagged for review? I don't really know how the algorithm works. enter image description here

If this is low-quality then I feel bad about all of my answers.

Here's a link to the question if you care to see it in context. It's not really a huge problem (as Bart's comment emphasizes), but this seems like an extreme outlier in every measurable aspect.

share|improve this question
11  
All computer algorithms that implement heuristics of this kind are occasionally going to give false positives, no matter how good they are. That's why there's a Review process. –  Robert Harvey Aug 10 '12 at 17:23
5  
"Looks Good"...done... (though perhaps the username triggered it? ;) ) –  Bart Aug 10 '12 at 17:24
2  
@RobertHarvey Sure, but this one is a bit too far off to not raise an eyebrow on meta. –  Daniel Fischer Aug 10 '12 at 17:35
3  
I wonder if this is a FGITW answer: started out in low quality territory, which triggered the flag, but the user managed to get the rest of their answer in before the 5 minute grace period ended. –  user149432 Aug 10 '12 at 17:46
    
@MarkTrapp, good point... Does it work like that? Actually I just looked at the question and this answer came around 2 hours after the first answer, so it seems unlikely. –  Dustin Aug 10 '12 at 17:51
    
@Dustin I don't know when the low quality flag is triggered: if my hypothesis is correct, it would mean the flag is triggered upon submit, and might need to be adjusted to trigger after the 5 minute grace period. But this might just be a really edgy edge case and this doesn't happen all that often to warrant that adjustment. –  user149432 Aug 10 '12 at 17:53
    
I think what might have caused this one to get flung into the queue are the two exclamation points in the text. Exclamation points are normally a big red flag, especially if they're consecutive. –  Robert Harvey Aug 10 '12 at 17:55
    
@MarkTrapp It seems like questions are removed from the low-quality review when they're edited to be better. Otherwise I feel like I'd be coming across "good" answers pretty frequently. –  Dustin Aug 10 '12 at 17:57
1  
@RobertHarvey It does make sense to look for those I guess... but there should be some compensation factor with the length of the question & the code samples. If this answer was shorter/didn't have code samples it would make sense to flag it for exclamation points. Again, I have no idea how the system works. –  Dustin Aug 10 '12 at 18:01
2  
What about an accepted answer with 10 upvotes? I really consider voting "Looks Good". –  Bo Persson Aug 10 '12 at 20:02
2  
Couldn't have been a bad answer if it had lists. –  Asad Nov 30 '12 at 8:52

1 Answer 1

up vote 87 down vote accepted

You caught me.

We throw a few known cases into the queue* as "gotchas" to keep people from just nailing "Looks Good" or "Close" or whatever without reading the questions.

If you see one of these just handle it normally, we're probably going to make them a little more... subtle over time. For now we're pulling from really obvious sources: spam, and uncontested high quality posts.

In the long term we're probably going to roll a "review score" up, and these "audit" tasks will be a small component of it.

*Note they aren't "really" there, you can't delete or close or whatnot (you don't spend a vote or anything either). You get a bit of a talking to if you choose the wrong option as well.

share|improve this answer
26  
ARE YOU SERIOUS –  Dustin Aug 13 '12 at 20:51
33  
That sounds like a GREAT idea; to keep people from just smashing the same button to just "go through" them... like the problem we have with suggested edits... –  Andrew Barber Aug 13 '12 at 21:04
    
@AndrewBarber I'm not sure I agree. Given that there are 35K+ low-quality posts on SO for review, do we really need extra work made for us? That said, if they're doing it just for the time being so they can see if it will be effective, that's ok, but throwing stuff in later that's obvious is just making work where work doesn't have to be made. Kind of like Community flags (but I won't go there)... –  casperOne Aug 14 '12 at 17:10
8  
@casperOne - this is done "on demand", audit tasks don't exist if real ones don't. While in theory it's possible for the queue to be nothing but audit tasks (requires people working through the queue, detecting audits, and intentionally taking no action) it would require a dedicated group of users intentionally doing that, with a scale of collusion that's probably impractical. If it actually happens we can pretty easily remove audit tasks once they're too common, using the same mechanism we do to remove questions that get quality improving edits from outside the queue. –  Kevin Montrose Aug 14 '12 at 17:14
2  
@KevinMontrose That's more reassuring, now stop clicking the audit button! –  casperOne Aug 14 '12 at 17:18
    
Is this done only or SO, or on all sites in SE network? –  Martin Aug 18 '12 at 12:51
1  
@Martin This is currently only enabled on SO, it will eventually be on everywhere. –  Kevin Montrose Aug 18 '12 at 17:53
1  
i find the fact the team is doing this reassuring. I always hoped such studies were done. As I hope you record and study other signs of bad reviewing (like people voting for or against closing as duplicate questions without opening them). –  dystroy Nov 30 '12 at 9:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .