TL;DR Does it make sense to establish (or at least test) new review queues at sites other than Stack Overflow?

As far a I understand, answer to above question is not quite straightforward:

This is all pretty much tailored to the needs of Stack Overflow... Not that this precludes using it elsewhere - but that's a discussion we'll need to have separately.


Given past experience, one can expect successful "expansion" of new queues network wide. As far as I know, all prior kinds of review queues have been successfully instilled to all Stack Exchange sites and were proven to be useful.

Still, there could be things to take care of there.

In particular, I would like to learn whether moderator actions in new queues are binding or not, and if not, then how this could impact sites with review activity lower than at Stack Overflow. (I think I have seen non-binding actions mentioned at MSO but could not find this in triage and help-improvement tags over there.)


Readers already familiar with new queues can skip the rest of the post, as there is only a brief overview of these.

  • Details of Triage review are laid out in this post at MSO:

    tl;dr: there's a new review queue. It'll be getting somewhere around 1-2 questions per minute. The only thing they have in common is that the system is unsure of what to do with them. Some are great, some are awful, some are in-between. We need you to help the system decide which category these questions belong in...

    Behind the scenes, a "quality score" is calculated for each question based on an automated analysis of the content. Those that score well are sent immediately to the homepage; those that score poorly will now be sent to Triage. From there, they'll go to one of three places based on human input:

    1. The homepage, where they can be answered
    2. The close or moderator flag queue where they can be reviewed and eventually deleted
    3. A new "Help and Improvement" queue where they can be edited
  • The new Help & Improvement Queue...

    ...is a place for anyone that has the desire to help new users learn the ropes in the best way possible - by jumping in and taking some ownership in their experience as you teach them what makes questions great through your edits and helpful comments. Remember, questions in this queue have been selected because enough people thought they showed strong potential, but needed some work before they were ready for full visibility.

  • The way how new queues integrate into the site and with each other is laid out here:

    All questions flagged Very Low Quality now immediately enter Triage...

    ...and while these questions are being triaged, they're not shown on the home page.

  • 1
    "not shown on the home page..." -- my quote breaks here in order to keep things brief but those who won't visit source post may enjoy a slightly expanded version: "... not shown on the home page. Take that, wall of crap." :) – gnat Mar 20 '15 at 18:18
up vote 18 down vote accepted

I'm glad you asked this. And I say this even though I can't answer it, which is unusual for me - I'm a narcissist and it really bugs me when I can't use someone else's question to show off my own smug self.

But I am still a narcissist, so I'm gonna write like a page of explanation for why I can't answer this first and only then speculate on what the answer might be; feel free to skip to the last heading.

Big city, small town

See, the truth is right now we don't even know if this stuff will work on Stack Overflow! And that's the whole point of doing it: for years, we kinda hamstrung our ability to fix problems on Stack Overflow by trying really hard to make every change work across the entire network, from Stack Overflow with thousands of posts a day all the way down to Sustainable Living which gets about one question every couple days...

That was a nice idea in theory. But it really wasn't fair to anyone; there are lots of tools that are complete overkill on smaller sites, and behaviors that don't even make sense to folks who aren't used to dealing with insane numbers of questions. Trying to even discuss these problems on MSE was getting to be a headache; some discussions involved more culture shock than anything constructive, as folks from tiny sites tried to extrapolate their small personal disagreements to SO's scale and SO regulars tried to apply the policies that were necessary there to folks who didn't so much need a rule as they needed to go somewhere else and just talk to each other.

So the first step toward fixing some of the pervasive quality issues on Stack Overflow was simply giving Stack Overflow a place to talk to itself, about itself.

A little bit of introspection goes a long way sometimes... The first two hugely-popular threads on the new MSO were focused on negativity and quality. Not new topics by any means, but indicative of a huge amount of pent-up frustration on both fronts. And most telling, folks saw these problems as representing conflicting goals, intractable by nature because trying to solve one would inevitably make the other worse.

A few people got really bummed about these discussions, but I think they're great. Finally, we had reason to stop blaming a few malcontents and start fixing underlying issues. Instead of shooting in the dark and hoping for the best, we had brightly-lit targets to aim at...

Two really hard problems

I'm gonna try & sum up the two goals of the quality project in one sentence:

Get a LOT more crap out of the way a LOT faster, while pairing up folks who CAN and WISH TO learn with those who CAN and WISH TO teach.

And here's the punchline: we're trying to accomplish both goals without pitting them against each other. Massively de-emphasize questions from folks who haven't yet learned how to ask a coherent programming question, but don't immediately nuke them - instead, provide the asker with gentle, general-purpose advice and offer anyone willing the opportunity to step in and give them some more specific guidance.

Sound impossible? Unlikely? Well... It might be. But we're trying anyway, dammit, because it's important!

And what you're seeing so far - the new review queues, that crazy flowchart - those are just the tip of the iceberg. We're building infernal machines deep underground for classifying content and mixing metaphors. They're big and scary and we probably wouldn't be getting away with it if we hadn't thought to paint "ice cream for children" in big colorful letters on the sides of the housings ahead of time...

...and we won't know if any of it really works until it all comes together. We're testing bits and pieces, a little bit at a time, and seeing what happens, what assumptions are valid and which ones aren't. It's nerve-wracking and tedious and we're using people as guinea pigs and...

...and so, you want to be a guinea pig?

This is the bit where I try to answer your question without actually answering it.

The Help and Improvement queue could actually stand alone. That'd probably be the first thing we'd consider turning on anywhere else, just to see what folks did with it. Of course, we'd have to have a different method of feeding it, but on smaller sites it'd actually be feasible to just throw in every 0-scored question from every user with less than 10 reputation and call it a day.

The content classifiers that feed Triage (and, indirectly, Help and Improvement) require training - which means we need a pretty substantial amount of known-good and known-bad posts. It'd be feasible to run on some sites, but not nearly all.

Triage itself is probably pointless on all but the very largest sites. The whole point is to chew through a huge number of questions as quickly as possible; if you're only getting a few dozen questions per day, it's unlikely that there are enough reviewers to make it work or enough questions to keep them interested in the first place. Aside: the only actions in Triage that are binding for moderators are those that are binding everywhere else: closing, deletion and spam/offensive flags - this means a single moderator can't just clear out the queue if no one else is around.

Finally, the altered Very Low Quality flag behavior only makes sense when both Triage and Help/Improvement are available. It's sorta what ties them together - without those queues and enough active reviewers to make them work, you're just forwarding all VLQ flagged questions straight to the moderators, which... is what already happens.

In short, I'm not yet sure if these queues make sense anywhere else. My gut feeling is that they'd be useful in some form on a few of the biggest sites - Super User, Ask Ubuntu, Mathematics, maybe even Server Fault if they can resist the urge to comment instead of editing... But first we have to make them work on Stack Overflow. If we can't do that... There's really not much point in trying to continue this experiment elsewhere.

  • 4
    I haven't yet completed studying of your answer but "Big city, small town" section alone is worth an upvote. Even if I later find myself in a total disagreement with the rest, thanks for that part. An awesome explanation of why things eventually started feeling so... right after mso-mse-split - both at MSO and MSE – gnat Mar 20 '15 at 19:10
  • "Help and Improvement queue could actually stand alone..." -- when (if) you turn it on at Programmers, consider dropping a line into Where and how to improve questions? over there – gnat Mar 20 '15 at 19:22
  • as far as I can see, site needs to cover three points for Triage to make sense: 1. set the goals (for SO, goals are said to be "de-emphasize questions from folks who haven't yet learned... but don't immediately nuke them" and protect home page from "wall of crap") 2. enough content to keep scoring algorithm relevant - 100 troublesome questions a month won't do, nor probably 1000 would. 3. Timely escape - we don't want automatic scoring nor user flags to hide question for hours solely because there's nobody around to review. Does that make sense? – gnat Mar 21 '15 at 10:53
  • 1
    Roughly correct, @gnat. To that I would only add, it must be clear to many that the existing tools are insufficient: in particular, if the "home page as work area" strategy is working well for a given community, there's little need for anything more elaborate. – Shog9 Mar 21 '15 at 17:26
  • I see, thanks. Your addition makes good sense - from what I learned so far, Triage looks like "weapon too heavy" to use it without clear and strong need – gnat Mar 21 '15 at 19:20

Well, for Photo-SE at least, I don't think this will help much. We basically already treat the Low Quality Posts queue as Help and Improvement, and don't have the volume where a more complicated workflow seems likely to be better. I expect that this is also the case for other lower-traffic but active sites.

  • LQ queue as H&I, how's that? The way that H&I was presented at MSO suggests something... rather opposite to Low Quality: " questions in this queue have been selected because enough people thought they showed strong potential..." – gnat Mar 20 '15 at 18:58
  • I guess we generally try to look for potential in most ... troubled ... new questions and answers, as long as they're not spam or trolling. With relatively low question volume, the front page works just fine for questions which show strong potential. – mattdm Mar 20 '15 at 19:00
  • so, you kind of put a lot effort into pulling (dragging, squeezing...) out the best you can from these? Aggressive Edits and stuff like that, correct? – gnat Mar 20 '15 at 19:13
  • 3
    Sometimes? But usually it's more a matter of a bit of back and forth in the comments and smaller edits to help the questioner learn to get the most out of the site — help, not just improvement. Other times, a small edit can make a big difference. (I can't remember if this one was flagged as low quality or just getting downvotes and close-as-unclear votes, but now it's in our top-20 upvoted questions.) – mattdm Mar 20 '15 at 19:26
  • oh, so tight interaction with asker is also an option, understood. That looks really much more than H&I offers, I think I understand now what you mean saying you don't need that queue. (side note edit you referred to looks fairly smart, I agree - not surprising if it made a turning point, despite small "char count") – gnat Mar 20 '15 at 19:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .