65

Today I came across an OBVIOUSLY good answer in /review. It had 7 upvotes, contained lists, paragraphs of explanation, and well-applied sample code. It was also moderately long. How does a question like this get flagged for review? I don't really know how the algorithm works. enter image description here

If this is low-quality then I feel bad about all of my answers.

Here's a link to the question if you care to see it in context. It's not really a huge problem (as Bart's comment emphasizes), but this seems like an extreme outlier in every measurable aspect.

Improve this question
11
  • 12
    All computer algorithms that implement heuristics of this kind are occasionally going to give false positives, no matter how good they are. That's why there's a Review process.
    – user102937
    Commented Aug 10, 2012 at 17:23
  • 5
    "Looks Good"...done... (though perhaps the username triggered it? ;) )
    – Bart
    Commented Aug 10, 2012 at 17:24
  • 4
    @RobertHarvey Sure, but this one is a bit too far off to not raise an eyebrow on meta.
    – Daniel Fischer
    Commented Aug 10, 2012 at 17:35
  • 4
    I wonder if this is a FGITW answer: started out in low quality territory, which triggered the flag, but the user managed to get the rest of their answer in before the 5 minute grace period ended.
    – user149432
    Commented Aug 10, 2012 at 17:46
  • @MarkTrapp, good point... Does it work like that? Actually I just looked at the question and this answer came around 2 hours after the first answer, so it seems unlikely.
    – Dustin
    Commented Aug 10, 2012 at 17:51
  • @Dustin I don't know when the low quality flag is triggered: if my hypothesis is correct, it would mean the flag is triggered upon submit, and might need to be adjusted to trigger after the 5 minute grace period. But this might just be a really edgy edge case and this doesn't happen all that often to warrant that adjustment.
    – user149432
    Commented Aug 10, 2012 at 17:53
  • I think what might have caused this one to get flung into the queue are the two exclamation points in the text. Exclamation points are normally a big red flag, especially if they're consecutive.
    – user102937
    Commented Aug 10, 2012 at 17:55
  • @MarkTrapp It seems like questions are removed from the low-quality review when they're edited to be better. Otherwise I feel like I'd be coming across "good" answers pretty frequently.
    – Dustin
    Commented Aug 10, 2012 at 17:57
  • 1
    @RobertHarvey It does make sense to look for those I guess... but there should be some compensation factor with the length of the question & the code samples. If this answer was shorter/didn't have code samples it would make sense to flag it for exclamation points. Again, I have no idea how the system works.
    – Dustin
    Commented Aug 10, 2012 at 18:01
  • 2
    What about an accepted answer with 10 upvotes? I really consider voting "Looks Good".
    – Bo Persson
    Commented Aug 10, 2012 at 20:02
  • 2
    Couldn't have been a bad answer if it had lists.
    – user200500
    Commented Nov 30, 2012 at 8:52

1 Answer 1

Reset to default
93

You caught me.

We throw a few known cases into the queue* as "gotchas" to keep people from just nailing "Looks Good" or "Close" or whatever without reading the questions.

If you see one of these just handle it normally, we're probably going to make them a little more... subtle over time. For now we're pulling from really obvious sources: spam, and uncontested high quality posts.

In the long term we're probably going to roll a "review score" up, and these "audit" tasks will be a small component of it.

*Note they aren't "really" there, you can't delete or close or whatnot (you don't spend a vote or anything either). You get a bit of a talking to if you choose the wrong option as well.

Improve this answer
8
  • 29
    ARE YOU SERIOUS
    – Dustin
    Commented Aug 13, 2012 at 20:51
  • 35
    That sounds like a GREAT idea; to keep people from just smashing the same button to just "go through" them... like the problem we have with suggested edits...
    – Andrew Barber
    Commented Aug 13, 2012 at 21:04
  • @AndrewBarber I'm not sure I agree. Given that there are 35K+ low-quality posts on SO for review, do we really need extra work made for us? That said, if they're doing it just for the time being so they can see if it will be effective, that's ok, but throwing stuff in later that's obvious is just making work where work doesn't have to be made. Kind of like Community flags (but I won't go there)...
    – casperOne Mod
    Commented Aug 14, 2012 at 17:10
  • 8
    @casperOne - this is done "on demand", audit tasks don't exist if real ones don't. While in theory it's possible for the queue to be nothing but audit tasks (requires people working through the queue, detecting audits, and intentionally taking no action) it would require a dedicated group of users intentionally doing that, with a scale of collusion that's probably impractical. If it actually happens we can pretty easily remove audit tasks once they're too common, using the same mechanism we do to remove questions that get quality improving edits from outside the queue.
    – Kevin Montrose
    Commented Aug 14, 2012 at 17:14
  • 2
    @KevinMontrose That's more reassuring, now stop clicking the audit button!
    – casperOne Mod
    Commented Aug 14, 2012 at 17:18
  • Is this done only or SO, or on all sites in SE network?
    – Martin
    Commented Aug 18, 2012 at 12:51
  • 1
    @Martin This is currently only enabled on SO, it will eventually be on everywhere.
    – Kevin Montrose
    Commented Aug 18, 2012 at 17:53
  • 2
    i find the fact the team is doing this reassuring. I always hoped such studies were done. As I hope you record and study other signs of bad reviewing (like people voting for or against closing as duplicate questions without opening them).
    – Denys Séguret
    Commented Nov 30, 2012 at 9:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .