# Let's get rid of the 10K flag queue

The 10K tools are pretty cool... You get a birds-eye view of activity on the site, a "dashboard" view of what's happening. Some of the individual tools haven't scaled particularly well with Stack Overflow's growth, but the concept behind them is still sound: we trust you to enough to be a bee watcher now.

...and then there's the flag queue. What a let-down!

Once upon a time, this queue contained spam and offensive flags, which one might reasonably assume were important enough to put in front of the more trusted members of the site. Re-flagging them brought them that much closer to deletion, while editing them offered salvation to some hapless author.

Nowadays, it's a bunch of Not an Answer flags and a smattering of assorted cruft. 10K users can't even vote to delete these; only 20K users have that privilege. Re-flagging them does nothing but increase their priority in the moderator flag queue, where they frequently outrank more pressing issues; disputing them has its own issues. Also, it's full of bugs, and the behavior has diverged far enough from that of the moderator flag queue on which it is based that it has become an active hindrance to further development of the tooling there.

Worst of all, we have a much better tool for handling crap answers that's available to anyone with the editing privilege. It even has logic built in to prioritize answers likely to be deleted for users with the reputation to delete them. It's like we gave you a car for your 10th birthday, and then replaced it with a rusty bicycle when you turned 16.

It was a nice idea, but it has outlived its usefulness.

## Proposed changes

The /tools/flagged route goes away. Period. No replacement. The rest of the 10K tools stick around as informational pages.

Not an Answer flags go into /review/low-quality, just like Very Low Quality flags already do.

Then we beef up the Low Quality review process to make better use of more experienced reviewers and solve this whole "declined / helpful / disputed" flag debate once and for all:

• Effective # of reviews required == ReviewsRequired + # of applicable flags (where ReviewsRequired is 2 on Stack Overflow, 1 everywhere else). So 1 VLQ or NAA flag means EffectiveReviewsRequired=3.
• LQ tasks are not dequeued until one of the following conditions is met:
1. Post is edited from within review.
Outcome: flags are marked "helpful" (current behavior).
2. Post accumulates 3 Delete votes (can only happen when post scores <= 0 and reviewers have >= 20K rep).
Outcome: post is deleted, flags are marked "helpful".
3. Task accumulates EffectiveReviewsRequired "Looks Good" reviews.
Outcome: if the number of (Recommend)Delete reviews is >= the number of Looks Good reviews, mark flags "disputed" and raise DisputedLowQuality mod flag. Otherwise, mark flags "declined".
4. Task accumulates 6* RecommendDelete + Delete reviews.
Outcome: mark flags "helpful". If the post scores > 0 then raise DisputedLowQuality mod flag, else just delete post (current behavior).
• As under the current system, flags on posts that've already completed one full review cycle without being deleted should skip /review and go directly into the mod queue.

### End result

• Flags get handled faster, more accurately, and with less wasted effort from 10K reviewers.
• Moderators are free to focus on situations that can't be handled by the community - the exceptional cases!
• Developers are free to make enhancements to the moderator tooling without having to work around 10K user restrictions.

## Questions

Am I forgetting anything here? I haven't really spent much time in the 10K flag queue since back when it was filled with spam flags and therefore useful - is there a use-case that you depend on that would be lost with its removal? Post 'em below.

*: on Stack Overflow, only 4 Recommend Deletion reviews are sufficient.

• I've never really understood the point of the 10k queue and used it because I assumed it must be helping "somehow" so I'd say you're not missing anything here – Richard Tingle Mar 20 '14 at 0:02
• We should just have a separate "duplicate question review" queue, @animuson - then we could feed those into that. – Shog9 Mar 20 '14 at 0:05
• Very nice and don't think you missed anything. But, while doing this change can you please make it that post deleted by 6 reviewers is properly marked as such? Current behavior is that only "Deleted" appears, without any name and there isn't even a delete event in the post revisions. Highly confusing. (I raised this a while ago) – Shadow The Vaccinated Wizard Mar 20 '14 at 0:11
• I don't spend much time in the low-quality review queue, because every time I go there I'm reminded of what's missing. I can't downvote, I can't reflag (e.g. to suggest a post notice, or because I've noticed a pattern and want to flag the user), and if my reaction is "meh" I have to either say "looks good" when it's not good, just ok, or do something with it, or (what I usually do) skip it (which doesn't help move it out of the queue). These are things I can do from the 10k flag queue, so while I agree that queue has a lot of issues, let's fix these LQ issues before killing 10k flags. – Monica Cellio Mar 20 '14 at 0:26
• @Shog, thanks for that pointer. I see the logic. I guess the key question is: what outcome(s) is the low-quality review queue supposed to produce? Correction through edits or deletion but not the space in between, downvoted? (I'm thinking out loud, not arguing a position, here -- I haven't thought about this problem as much as you have. I just know that in my own experience the LQ queue makes unicorns cry, or something like that, so I avoid it.) – Monica Cellio Mar 20 '14 at 2:09
• @MonicaCellio I kind of want to view the LQ review queue as a delete queue of some sorts, just decide what's good, bad and ugly, good stays, bad gets fixed and ugly removed. – Braiam Mar 20 '14 at 2:17
• I'm with @Monica on this one. Wouldn't it be nice if we had a way to improve poor answers prior to banishing them?. I was so excited when I saw this idea, and thought it would tackle the problem, and while this will make poor answers easier to delete (a good first step), it isn't giving answers a similar status to closed/on-hold which would be greatly appreciated on more subjective sites. – jmac Mar 20 '14 at 2:51
• I will hate this change. That's all I have to say. I agree with @MonicaCellio, this, like the recent closing changes (with regard to TL at least), was made to circumvent a problem, not fix it. That's not good. The 10k flag tool is quite useful on some of the medium sites, it really does help prioritize. I'm really sick of you removing whole features just because they don't work on SO. Stack Overflow is the exception, not the rule! – ɥʇǝS Mar 20 '14 at 3:01
• Post an answer, @Seth. Be specific about what you would miss were this implemented. Hand-wavy "this is awesome for some unspecified reason on some unspecified site" responses ain't gonna change anything. – Shog9 Mar 20 '14 at 3:09
• Can we have a Star Trek weapons console in /tools once all of this is done? I kinda like the weapons station on a Klingon Bird Of Prey and because cloaking device. – Tim Post Mar 20 '14 at 3:20
• @Shog, that's a different way of looking at the issue. Generally speaking I see content in three categories: good as-is, needs improvement, crap. Crap should be deleted promptly. Good as-is content should be allowed to thrive. 'Needs improvement' content should be given a chance before being tossed in to the 'good' or 'crap' bins and dealt with. We have this for questions (which have a queue, are auto-deleted if not dealt with, and easier to delete by the community), but not for answers. I want to know why answers should be tougher to deal with than questions. – jmac Mar 20 '14 at 3:44
• @Shog, "Needs work" is what close votes are for with questions. And that meant it should be deleted if not improved. So you made closing more friendly to improvement. This works for questions, why not for answers? You can tell me to downvote, and I do, and that's swell, but the problem remains: the system for dealing with poor answers is worse than dealing with poor questions which is one of the reasons for this change. – jmac Mar 20 '14 at 4:20
• To add to what jmac said ^^^, answers are the bread & butter of Stack Exchange; that's why we're here. We need better tools for dealing with poor ones. It's even too hard to flush the crap on small sites, to say nothing of the ones that could possibly be improved. Say I'm looking at an answer that I know needs work but I myself lack the relevant expertise -- what am I supposed to do with that? Not everybody who can spot a problem can fix it and almost nobody can vote to delete, so all we can do is ask mods to, um, maybe add a post notice, or try to rally trusted users in chat. Too much work. – Monica Cellio Mar 20 '14 at 13:20
• The SE model doesn't work well for sites that attract mostly subjective, unverifiable, opinion-based answers. Yet SE has such sites, and they can't rely on mods to deal with answer-quality problems (as you often say, not their job), so there have to be better ways for the community to act. Don't make it even harder, please. The 10k queue is bad in a lot of ways but does help with some problems. – Monica Cellio Mar 20 '14 at 14:19
• @Shog, "some existing sites need such rules and tools - but that's really a different discussion" -- when you are eliminating one of the tools that the community finds useful due to the lack of alternatives, then it would seem that the discussions are at least tangentially related and not separate at all, no? – jmac Mar 20 '14 at 15:14

This is done:

• /tools/flagged is no more. Disputed flags are still a thing, but disputes are situations observed by the system, not explicit actions. The guidance for new 10K users has been revised: https://stackoverflow.com/help/privileges/moderator-tools

• Not an Answer flags now feed into the Low Quality review queue. This was mostly implemented as described in the proposal above, but since that was written as an implementation guide here's a somewhat more readable summary of the rules:

1. VLQ and NAA flags immediately enter the mod flag queue in all cases
2. VLQ and NAA flags on posts that have not been previously reviewed and are not closed, deleted, locked or accepted will enter /review/low-quality-posts.
3. If review completes successfully, the flag is marked handled and removed from the mod flag queue
4. If all mod-flags on a post are dismissed from the moderator flag queue, the review task is invalidated
5. If the outcome of review is deletion, and a post's score prevents it from being deleted, the original flags will be marked "helpful". In these cases, and in cases where the flags are disputed, a new, moderator-only flag will be raised on the post.

We've been operating with these rules in place for about a week now. During this time, 1656 review tasks have been created and processed based on Not An Answer flags:

• 339 of these were resolved as Looks Good
• 1251 of these were resolved as Delete
• 122 "disputed low quality review" flags were raised as a result of conflicting reviews

A couple of observations can be made from this easily:

• 75% of the posts reviewed are considered worthy of deletion by reviewers. Adding in the number of "disputed" tasks puts this at ~83%, which is reasonably close to the 88% accuracy rate on NAA flags handled by moderators in March. More on the discrepancy in a bit.
• This has effectively reduced the "not answer" workload for moderators to less than 10% of what it would have been otherwise.

Now, about that discrepancy... We expected to see this dip a little bit: reviewers as a group are somewhat more tolerant of borderline answers than are moderators. In addition, there are a few other factors at work here:

• A backlog of several hundred NAA flags that hadn't been urgent enough to process before now and may have contained a larger % of borderline or straight-up invalid flags.
• As noted above, some of these flags are still handled directly in the moderator queue. 947 review tasks were invalidated due to the flags being handled outside of review, with 908 of the associated posts being deleted. This alone, added to the numbers above, puts the % of helpful flags very close to 88%.

Overall, the results so far have met or exceeded expectations. We'll continue to monitor this - please report any problems you observe.

• Has the definition of NAA changed at all then? Given the inability of the community at large to understand what NAA is, I'm kind of skeptical that reviewers can be trusted to correctly resolve these flags. I'm envisioning robo-reviewers seeing wrong/poor answers and deleting them as non-answers. Sure, poor answers should be deleted, but for the right reasons. Or is the thinking now that with hundreds of reviewers in the queue, the load of incorrectly cast flags can be borne by the community so long as the content that should be deleted gets deleted? – Jonathan Garber Apr 9 '14 at 18:05
• (I ask this from the perspective of a reviewer on a smaller site. We don't get as many wrongly cast NAA flags there, but they do happen, and they have been increasing in frequency over the last month.) – Jonathan Garber Apr 9 '14 at 18:06
• Lots of flaggers get that bit wrong, but reviewers tend to err on the side of "if it looks like an answer, it's an answer" - the most common mistakes tend to be those where someone phrases their answer in the form of a question or prefixes it with a backstory about their own problem. Hence the complicated logic for forwarding posts where reviews were mixed to moderators. – Shog9 Apr 9 '14 at 18:19
• Excellent. Correct handling by reviewers is what I was concerned about, so if it's actually working, that's a big plus for the review community in general. – Jonathan Garber Apr 9 '14 at 18:48
• Can we get stats on Community's handling of flags added at: stackoverflow.com/admin?filter=day perhaps, to get a feel for how things are ticking over in the background too? – Flexo Apr 10 '14 at 22:17
• Can we get a breakdown of the efficiency in smaller sites too? – Braiam Apr 12 '14 at 5:29
• On sites with enough data to be worth calculating this, the numbers are pretty close to what I have above, @Braiam. Sites that get ~1-2 flags a day are all over the map, but no one's reviewing there regularly anyway. – Shog9 Apr 12 '14 at 14:35
• I'm interesting how disputes are done now. How does this work? – ɥʇǝS Apr 22 '14 at 2:08
• It is described in the feature-request above, @ɥʇǝS – Shog9 Apr 22 '14 at 2:18
• Is planned the LQ-queue counter for >Xk rep users? – Braiam Apr 22 '14 at 13:57
• @Shog9 oh right. I did know that, I just forgot ;-) – ɥʇǝS Apr 22 '14 at 14:16
• I've just found this and I believe it's really unhelpful. Moderator flags are for moderators to deal with. At the very least, posts which are put into a queue should be marked in the mod-flag queue as "Queued for community action in VLQ queue" or something. (That is, we shouldn't have to know or divine that; and nor should moderators have to deal with something which is not an exception.) – Andrew Leach Sep 9 '14 at 11:14
• If it's not being handled in /review, it becomes a moderator problem @Andrew. – Shog9 Sep 9 '14 at 14:32
• But that means that it only becomes a moderator problem when it's not being handled in /review. It's not a moderator problem when it's put in /review. Perhaps a time could be added in the mod queue so that a judgement might be made as to whether it's a problem. – Andrew Leach Sep 9 '14 at 14:39
• Yes, we might do that eventually @Andrew - we had that in the past for the 10K queue, but dropped it because the 10K queue wasn't handling much of anything; review seems a bit better in this regard. – Shog9 Sep 9 '14 at 15:36

I've had a request like this on the back-burner for a while. In a nutshell:

• The 10k queue is bad because most of what is there cannot be dealt with by its public.
• Even when 10kers can contribute, they can rarely remove the item from the queue.

As a user on Stack Overflow and elsewhere, I used to participate a bit in the 10k queue back in the old days, but I stopped because I didn't seem to achieve anything there. The only flags that were useful were the close flags and the spam flags. Close flags thankfully moved to the close review queue which is a much saner approach, and spam flags are now shown to mods only (I assume this is because they were abused, which is a pity because fast community intervention would be better than having to wait for a mod to come along). Now, it seems to me that pretty much anything that's left is:

• either things that I can't deal with — I can help by commenting, but so can anyone, and I can help by investigating, but not much;
• or answers that are to be deleted, which requires 20k rep, not 10k.

Feeding close flags into the same close/don't-close queue as close votes was definitely the right thing. Similarly, delete flags (a.k.a. not-an-answer or very-low-quality — but there's no point in having two different flag names for the same effect) should feed into a single delete queue, together with posts with delete votes (thresholds may need to be adjusted accordingly). The point is to show people things that they can actually deal with, in such a way that when enough people act the thing is done without any moderator involvement.

As a moderator — but only on smaller sites, not on SO — I don't remember ever seeing anything useful come out of the 10k flag queue. Pile-on flags are useless: it's my job to review the flags — whether there are 5 NAA flags or 1 NAA flag and 3 “invalid flag” flags, I still have to make my own opinion.

Inasmuch as Community's flags are helpful — and they rarely are — dealing with them typically requires moderator intervention. Consecutive close questions? Check the user's deleted questions, resolve by maybe warning or suspending the user. Possible self-vandalism? Check the user's recent deletions, resolve by maybe warning or suspending the user and undeleting. Excessively long? Why is there even a flag for that? The only one that could be dealt with by non-moderators is “duplicate answers” (possible close as duplicate votes, and deletion of the answer); and this one often requires moderator involvement anyway to delete the answer, if it's been upvoted.

A couple of things I'd add to the delete review (or low quality review if you want to call it that way, but I think delete review is clearer, similar to close review):

• Make sure that there is a clear way to explain why the post should be deleted. Any comment left on the post should notify the user.
• Provide a formal way to indicate when an answer should be converted to a comment. Often flaggers use “not an answer” (as officially recommended), leaving the moderator to figure out again whether the answer should be removed outright or converted to a comment. Even when someone raises a custom flag, often that flag is not as helpful as it could be — flags like “please convert to comment on Bob's answer” (or worse, “please convert to comment on the top answer”…) are an unnecessary communication hindrance.

(Sorry for the length, I don't have time to make this post shorter.)

• I think a lot of this could be improved by allowing flaggers to add post notices to answers and then having the resulting answers that don't get fixed tossed in to the deletion queue. Since normal users (10k+) can actually delete things in the queue, many of these flags can be handled by the community through a sane workflow rather than having them be handled by mods. Not to mention it would give people a chance to improve their answers with clear guidance that there's something wrong beyond DVs. – jmac Mar 20 '14 at 2:55
• Communities flags are immensely helpful! I have no idea what you're talking about there. – ɥʇǝS Mar 20 '14 at 3:06
• @jmac All of this can be done by anyone (at least anyone who can comment, i.e. with 50 rep). What does the 10k queue add? – Gilles 'SO- stop being evil' Mar 20 '14 at 3:08
• @Seth Example? I usually decline Community's flags. Granted, that's because most of the ones I see are “closure without comment” flags on beta sites, the other ones are rare. But I rarely find the other ones useful either. – Gilles 'SO- stop being evil' Mar 20 '14 at 3:09
• @Gilles, I don't like the 10k queue for all the reasons Shog mentioned. It allows us to point out that there is poor content, but doesn't actually allow a bulk of the community to do anything about it. I want to allow normal users to take action on answers as they do on questions, since that allows the community to moderate content. I'm not disagreeing, just think it can be further improved to minimize moderator load and maximize the ability of the community to make those value-judgments on what content is right for their community. – jmac Mar 20 '14 at 3:42
• "Pile-on flags are useless" - Yep, pretty much. When there's 10 people flagging NAA and 10 people disputing the flag, I still have to make a very tough call... It took me actually experiencing this firsthand to really understand it... – jmort253 Mar 20 '14 at 3:58
• @jmac I, too, want normal users to be able to take action on answers, and that's precisely why I want to replace the 10k queue with a delete review queue (generalizing the current VLQ queue a bit). I don't see the point of post notices: a comment that explains the specific problem with the post is much more helpful. – Gilles 'SO- stop being evil' Mar 20 '14 at 10:55
• @Gilles, the comment thing has been discussed ad nauseum in the past and simply is never going to be enforced as mandatory. Post notices are good for when someone says, "You should do this" but doesn't bother explaining why or how that is the proper answer. So we can tack on a nice notice that says: "We're looking for long answers that provide some explanation and context. Don't just give a one-line answer: please explain why you're recommending it as a solution. Answers that don't explain anything will be deleted. See Good Subjective, Bad Subjective for more information." – jmac Mar 20 '14 at 15:26
• This not only signals to the user what is wrong with their post (and that it may be removed if they don't do something about it), but it also signals to everyone that reads the post notice that we have quality standards for content that doesn't include, "This is my opinion" one-liners that add nothing to our status as a resource. – jmac Mar 20 '14 at 15:27

The 10k flag queue has a lot of problems and I'd love to see it be replaced by a better low-quality-posts queue. There are some use cases that would be negatively affected by the proposed change, so I'd like to see a couple changes to the proposal.

The purpose of this queue (whatever form it takes) is to allow the community to act on poor content, improving it where possible and responding to it in various ways where not. Depending on the nature of the problem, a few different things can happen here:

• Sometimes a post is pure crap and the correct response is to delete it. (10kers can't do that, only request it.)
• Sometimes the reviewer can edit to fix the problem. Great, do that! But we can't assume that everybody who is expert enough to spot a problem is also expert enough to fix it, so "then just edit it" isn't a response that always fits.
• Sometimes it just needs a downvote (and optional comment) -- it's a bad answer but it's an answer and it complies with site guidelines. In this case the flag queue (or review queue) serves to bring bad posts to the attention of higher-rep users who missed them on the front page.
• Sometimes a post might be salvagable, but it requires the author or some other very small subset of users to do it (e.g. if it fails to cite sources but there's some reason to believe sources might exist). These posts should generally be annotated (with comment to ping), with some reminder mechanism so we can check back in a couple days. (Alternatively, I sometimes see these posts get deleted with a comment saying "flag for review once you've fixed it".) Annotations can only be added by moderators at present, so in this case the post needs to be re-flagged for moderator attention. And somebody has to remember to check back later.
• Sometimes the post is just fine. Not all flags have merit.

For all its problems, the 10k flag queue allows reviewers to vote, edit, comment, and flag. I don't spend much time in the low-quality review queue because I can't downvote, I can't flag, and if my reaction is "meh" I have to either say "looks good" when it's not good, just ok, or edit (but what if I personally can't fix it?), or (what I usually do) skip it (which doesn't help move it out of the queue). Processing this queue usually feels like a waste of time.1

These reviews are particularly important for answers. Answers are the bread and butter (or pearls) of Stack Exchange; that's why we're here. We need effective tools for dealing with poor ones. The 10k queue isn't a good tool, but it provides some capabilities that should be preserved in its replacement. Ideally we make the replacement even better, e.g. by allowing the community to vote on post notices, but let's at least not lose functionality.

People who spend most of their time on sites where answers are testable and are either right or wrong may not see the magnitude of the problem. Some sites suffer the double whammy of (a) being more subjective (answers are hard to verify) and (b) covering topics where everybody thinks he's an expert. Most of us work, so we're obviously qualified to answer any question on The Workplace. Any parent is obviously qualified to answer questions on Parenting. Anybody with some religious background is obviously qualified to answer questions about the bible. Or so the theory goes -- but it's wrong. Sites venturing into these fuzzier, subjective areas need to be able to weed out the bad answers, because they sure do get a lot of them. (I've got dozens more examples where those came from; I took the ones I had handy, which aren't necessarily even the best examples.)

The SE model doesn't work well for sites that attract mostly subjective, unverifiable, opinion-based answers. Yet SE has such sites, and their users can't rely on mods to deal with answer-quality problems (as Shog often says, it's not their job), so there have to be better ways for the community to act. And if not "better", at least "not worse than now".

I like the proposed move of the 10k flags queue into an improved low-quality review queue, so long as this new queue supports voting and flagging. As a stretch goal, please consider incorporating some way for the community to vote on post annotations.

1 I'm aware that I can always leave the review queue (or launch another tab) to do things not possible from the queue. That's a burden, though, and, demonstrably, one I often don't take on. We often hear that people don't downvote enough, in particular; we already have to get past the "oh no my precious 1 rep!" barrier, so let's not add another one for this in particular. Votes should be the primary community response to posts; make it easy.

• In theory, yes: the current flag queue provides a lot more in the way of tooling. In practice, it's still at least a click away (you have to expand the post to access it) and almost no one uses it. In the past 7 days, only 896 votes have been cast from the flag queue - that's across the entire network. If I exclude Stack Overflow, we're down to 322 votes. To put this in perspective, 24826 flags were raised from /tools during the same time period. Same problem as the old review system: lots of tools, no users. – Shog9 Mar 20 '14 at 20:57
• That's up to 896 more votes than would have been cast otherwise. (We don't know how many would have been cast through other means, of course.) When you talk about flags from /tools do you mean all tools (including stats, close, and delete), or just the flag queue? – Monica Cellio Mar 20 '14 at 21:01
• The flag queue is the only place where you can cast votes, @Monica - you have to click through to another page elsewhere. Technically, even voting from /tools/flagged is another page load (and the same # of clicks as it would be from /review), but since it's an AJAX load the referer is still /tools. – Shog9 Mar 20 '14 at 21:04
• Oh right; should have realized that. The AJAX load doesn't feel like a new page load (even if it technically is), so at least to me the click to add some inline context is very different from a click to go to a new page, @Shog. – Monica Cellio Mar 20 '14 at 21:07
• Maybe not to you, but it's still another click - and the vast majority of folks using the queue don't click. – Shog9 Mar 20 '14 at 21:08
• So is providing that affordance for those who do (and those who we might be able to encourage more, as we build community) expensive? – Monica Cellio Mar 20 '14 at 21:12
• Yes, extremely expensive. I'll elaborate more on this when I'm able to, but the short of it is that this page has been slowing or blocking badly-needed fixes to the (much more heavily used) moderator flag queue. So we can invest serious time in fixing a badly broken tool that sees relatively little use, or get rid of it and fix a critical tool. (slightly less than twice as many people using /tools/flagged as there are moderators, many times fewer actions taken as a result - it is effectively dead weight) – Shog9 Mar 20 '14 at 21:18
• I don't want to keep the 10k tool; see my first sentence. I want to keep the ability to vote and flag in the replacement queue. In the LQ review queue you're already showing the post (and IIRC its score); I'm asking for the vote buttons and the "flag" link. I like your proposal; it just doesn't go quite far enough. – Monica Cellio Mar 20 '14 at 21:25
• The cost of combining multiple reviews and voting - we could possibly 'fix' this by not making votes actually "count" as a review, but somehow I don't think that would satisfy anyone either. – Shog9 Mar 20 '14 at 21:28
• Huh, interesting. How did your answer over there work out in actual use? (And while this would probably be terrible UX, I'll just throw this out there: for the LQ review queue it's sufficient to enable downvotes. I guess if that were one of the orange buttons, alongside "edit" and "looks good" etc, rather than up/down arrows, that might solve the UX problem.) – Monica Cellio Mar 20 '14 at 21:33
• That would be terrible UX, and rather unfair. We did mock this up back when we were designing these queues, but discarded the idea. Something just occurred to me though - not all "votes" are actually votes. So I went and broke down those 896 votes by vote type: 629 delete votes, 184 down votes, 68 up votes, 9 favorites (yes, really), 2 undelete votes, and... The rest I'm not sure; possibly someone screwing around. Anyway, that's 252 actual votes over 7 days across the entire network. – Shog9 Mar 20 '14 at 21:42
• To put that in perspective, I tracked 114 votes from Low Quality reviewers clicking through to the question page, just on Stack Overflow in just the past 22 hours. Really don't think we're losing anything here. – Shog9 Mar 20 '14 at 21:47
• Interesting data. (So I'm, like, 5-10% of the network-wide vote-from-flag-queue hits? Wow.) SO is kind of a different beast from other sites; per-site breakdowns of actions from the 10k and LQ queues for the sites that are reporting problems might be interesting. (I wonder what this looks like on Workplace.) – Monica Cellio Mar 20 '14 at 21:53
• 20 votes from /tools on The Workplace in the past week. 3 of them yours. Only 4 users involved total. Probably worth noting I'm pulling all these numbers straight out of the logs, so there's no "debouncing" going on - if one of those voters happens to vote-unvote-vote-unvote-vote, that actually gets counted as 5. And yes, folks do that. – Shog9 Mar 20 '14 at 21:55
• I'm surprised that you find the VLQ queue to be a waste of time and the 10k flag queue useful. I have the opposite experience, both on SO and smaller site: the 10k queue is full of stuff that I can't remove from the queue, so I'm wasting my time, whereas the VLQ queue lets me participate in dealing with the item. – Gilles 'SO- stop being evil' Mar 22 '14 at 0:01

I'm all for the restructuring of the 10k tools. That used to be a place that I found a lot of value before I was a moderator, but the things that I found of value there (a list of questions with 3+ close votes, access to Late Answers and First Posts, etc.) now have new homes in the review system.

Frankly, the "Not An Answer" list of flags is often being abused now as a means for people to farm helpful flags (which really got bad with the Winterbash hats and around moderator elections), and I don't get a lot of value out of the number of people flagging something as "Not An Answer". When you've seen good answers with 30+ "Not An Answer" flags on them, you take all of that with a grain of salt. One flag is enough.

However, I want to expand on Lance's answer, because I think he brings up a very good point. If we're examining how to improve the workflow behind handling non-answers, I think it helps to look at root causes behind these answers. Ignoring spam and gibberish, almost all non-answers are attempts to comment on something.

The point of the reputation bar that a user has to clear is to prevent comment spam, which would be a very real problem if everyone could comment. However, I believe that if we had better tools to moderate comments in the same way we can questions and answers, it would be possible to watch over and deal with problematic comments from new users.

I've run this past other moderators, but I believe an equivalent for the First Posts and Late Answers review queues could be applied to comments as well. For moderators (and potentially 10k+ users) we could expose a list of recent comments, and make comments searchable to look for patterns of spam. We could add a new category of comment flags for spam and offensive content, weighted in the moderator tools like current spam flags.

With those tools, and with some of the limitations expressed by Tim in his comment on Lance's answer, I believe we could open up commenting to 1-rep registered users and significantly reduce the frequency of non-answers. Yes, you'll see more chatty comments come in ("me too, I'm having this problem", etc.), but the community will be able to flag those for removal just as we can non-answers.

Overall, I think that would make for a much less frustrating experience for newer users visiting the site, hopefully leading to some of them converting into more active members, and reduce the amount of work needed to keep the site clean. It also would give us the means to deal with spammers who somehow pass the current reputation threshold for leaving comments (which has happened, and we had no means of detecting or managing this).

• "When you've seen good answers with 30+ 'Not An Answer' flags on them, you take all of that with a grain of salt." Relevant – BoltClock's a Unicorn Mar 20 '14 at 16:33

Now days, it's a bunch of Not an Answer flags and a smattering of assorted cruft. 10K users can't even vote to delete these; only 20K users have that privilege. Re-flagging them does nothing but increase their priority in the moderator flag queue, where they frequently outrank more pressing issues

So the issues are:

1. The community can't actually handle the flags there
2. Taking the time to re-flag doesn't actually do anything about the content
3. The entire process screws with moderator priority

Eliminating the queue, while certainly the right thing to do, will not solve the problem by itself, it will just shift the problem to a different queue which may be better suited to deal with the problem, but still won't solve the actual cause.

1. Why can't most users handle the flags? Because the flags are on bad answers.
2. Why doesn't the flagging solve the problem? Because the flags are designed to identify content, not improve it
3. Why does this screw with moderator priority? Because poor answers should be handled by the community, like questions

Questions have a far smoother workflow with 1,657k reviews for 6,909k questions (23.9%), while answers have only 661k reviews for 12,174k answers (5.4%).

Without any good way to deal with poor answers, people raise flags about them (Not an Answer, and Very Low Quality, both which end up in the flag queue). And those flags just end up passing the buck to the moderators. So to fix the problem, we should attack the source.

## The tenets of community moderation

There are three types of posts. The community needs a way to distinguish between the three to take the appropriate action (or to allow the mods/system to take appropriate action):

1. Good -> No Moderation Required
2. Needs Improvement -> Give Guidance/Time to Improve

Where answers are getting stuck is on that second point. When the close votes were changed last year, here is what the SE team said:

Now, it’s not that we want those questions, but we need to convey exactly why we don’t want them. Imagine if police could give out summons that, rather than, “failure to stop at a signal,” just read, “behavioral violation”. When feedback isn’t specific, it’s impossible to fix the problem, but easy to write it off as probably coming from a bunch of grumpy old jerkfaces who’d rather make you look like an idiot than actually help you.

There is no way for the community to do this save through comments. While we are encouraged to add comments when we downvote, it certainly isn't mandatory (much to the chagrin of the folks being downvoted).

Not to mention that in addition to not explaining how to improve the post, we don't even explain if it can be improved. Some answers should just be removed immediately, such as:

• People posting a question as an answer
• People saying 'I have this problem too' as an answer
• People posting a comment as an answer

For answers that can be improved, there is absolutely no system to see if they have been improved. Again, from the change to close vote reasons:

The goal was always for some closures to drive an edit, improve, re-open cycle. The user gets helped, gets better at asking, and the community gets useful content. Unfortunately, since there was no way to know when a question had been improved, this almost never happened.

The same has been said about answers but has been marked because we are told that @notifications are good enough if people leave comments.

## Dealing with poor answers requires more individual effort than questions

The close system has been around forever. Here's a description of its purpose:

When you vote to close a question, you are really voting for that question to be eventually deleted ... a closed question is no longer alive in any meaningful way, and certainly well on its way to the bit-bucket of /dev/null.

We have since revised that to say, "so long as nobody tries to improve it within 5 days", but the base functionality is the same. Bad content gets closed, and then it will be dealt with by the system (either reviewed and reopened if improved, or auto-deleted if it isn't).

It doesn't work that way for answers. I can downvote, but will never be informed if it's changed unless I check it myself or am @notified. If it isn't improved, it won't be auto-deleted by the system, and unless it collects many downvotes, it won't even be reviewed for deletion by the community. The only exception is if I have over 20k reputation and actively look for answers to delete.

Close votes are free, downvotes aren't (which means people are often reluctant to downvote). Close votes are clear that you think the content isn't good in its current state and gives it a chance to become good, or be discarded. Downvotes are no substitute for close votes, and I don't think anyone would advocate we eliminate the close vote system entirely and rely on downvotes, comments, and @notifications for community moderation of question content.

## Allow the community to moderate answers

While I have a separate feature request, I'm not hung up on the details. The fundamental issue is that the community is not being given the tools to deal with poor answers. Since we do not have the tools, we end up creating additional burden for the moderators not because we want to, but because we just don't have anything else we can do.

What would the flag queue look like if the community could do the following?

2. Convert questions-as-answers to separate questions

We need a tool to allow us to do with answers what we do with questions. That is what the moderator privileges are all about -- giving us the tools to help moderate more given our rep. Giving us the right tools will allow us to give feedback on answers that allows actionable improvement on the part of askers, and it will allow the system to identify what posts are problematic and need revisiting if edited, or removal if not.

Shifting the Not an Answer/Very Low Quality flags to the Low Quality queue may reduce the moderator burden and increase the ability of the community to handle some of these flags, but it won't solve the fundamental problem: the community has no workflow to deal with poor answers.

the current tool has only one benefit which it was likely never intended to: when reviewing the flags, I find nice places to deposit my downvotes. This increases the chance that these answers will end up in the VLQ queue and can be deleted. This is not the intended behavior, but it does currently work. So long as downvotes are rare, and upvotes from passersby are common, the VLQ queue will remain sparse and underused.

When we get poor answers, someone will often flag it on TWP. That adds that nice blue moderator tools box to our top bar and increases the eyes of high-rep users on those poor answers. While the flags themselves may not help anything, it does bring our attention (and downvotes) to those answers. If we have enough high-rep users around, this will often push an answer down below zero (quite a rarity on our site), and actually get it tossed in to the VLQ review queue as a result.

We have just recently (past month) started having more active flagging, and it has had a large impact on the size of the VLQ queue. Since the data explorer doesn't actually track the data, I can't give exact numbers, but for reference we have had 531 reviews to date. Today alone we have had 21 reviews, and my guesstimate would be that the amount of NAA/VLQ flags issued correlates pretty well with the number of flags issued on answers.

I do not think this is a good reason to keep the queue around, the result is good but it doesn't mean the system is. But it is something to bear in mind. If this 10k flag queue is eliminated, please watch the more subjective sites to see if there is a dip in downvoting or items in the VLQ review queue after the change.

• I don't see what's missing in the community's ability to deal with poor answers, other than the current difficulty of getting a post through the VLQ queue (can't do it if it's upvoted, aborts if a single person disagrees). I don't participate on any site that uses post notices; I've never seen the point. On the budding Software Recommendations site, if an answer doesn't meet our minimum standards, we delete it, normally with a comment that points to the specific problem with the answer (and not some generic “you suck, RTFFAQ” notice). – Gilles 'SO- stop being evil' Mar 20 '14 at 10:59
• You're very fixated here on a relative minority of answers that... Well, that clearly and obviously do answer the question but are lacking in some way. I'm not at all convinced that these are a particularly good fit for community review, since by definition there's no consensus on whether they're good or bad! I elaborate on the potential pitfalls of such a system here, but I'll add here that most answers as comments make for lousy comments so "convert" is just kicking the can. – Shog9 Mar 20 '14 at 14:38
• @Gilles, the issue is getting a question IN to the VLQ review queue in the first place. We have a lot of really bad answers on The Workplace because we get a higher volume of answers compared to sites like stack overflow. And people like upvoting stuff far more than downvoting. Positively scored posts never get to the VLQ queue in the first place. Normal users with under 20k rep can't delete answers. This is a big problem when you only have 4 users >20k. – jmac Mar 20 '14 at 15:19
• @Shog, we are getting plenty of discussions on what constitutes answering the question and even that is up for debate. See this meta question on TWP. When someone asks, "How can I make my team come in on time?" people answer "You shouldn't do that" and they get hundreds of upvotes because people don't want to be forced to come in on time. SE wants to be different from forums, so give us the ability to actually make that a reality for subjective topics. – jmac Mar 20 '14 at 15:23
• So who decides which answers deserve votes, @jmac? – Shog9 Mar 20 '14 at 15:33
• @Shog, on sites where the users with the problem are most likely to be the voters (like SO, theory, no data to back it up), it's absolutely no problem. For the smaller sites with less daily issues covered (how often are you going to change jobs?), it means far more people get the right to vote through association than any specific expertise or actually facing the same problem and having to solve it. This makes the voting more 'noisy'. After all, I don't think many people will read through random [google-charts] questions and upvote their favorite answers, but they do the equivalent on TWP – jmac Mar 20 '14 at 16:00
• (note: I realize that with many tags on SO, this isn't how it works, and many tags are larger than the entirety of a site like The Workplace, but you generally have a community of experts or people with the problem issuing the votes, rather than people passing by for entertainment or because they want to waste 5 minutes before taking lunch) – jmac Mar 20 '14 at 16:01
• Regarding your edit: the proposed change builds your goal into the system explicitly by cutting out the middleman: flags go directly in /review. Of course, you can (and should!) still vote, comment, edit, consider whether or not the question itself needs work... But you should do all of that in the context of the question with all of its answers - neither /review nor the flag queue encourages good holistic moderation. – Shog9 Mar 20 '14 at 19:04
• @Shog9 evaluating an answer's quality should depend only on that answer and the question. Evaluating an answer's possibly redundancy depends on seeing the other answers, but that's not what reviews are for. Shouldn't each answer stand on its own? – Monica Cellio Mar 20 '14 at 20:25
• Regardless of what the platonic ideal should be here, the truth is that answers don't stand alone @Monica: they stand under a question and next to other answers. That's how they're viewed by the people who find them, y'know, with actual problems to solve - therefore, attempting to rank them in some artificial environment that strips them of context is bordering on dishonest. As I noted before, we allow it in very specialized circumstances carefully constructed to minimize the potential damage - but frankly, the results even there haven't been great. – Shog9 Mar 20 '14 at 20:29
• @Shog9 if that were really the case, then I'd be able to change a previously-cast vote in light of newer answers. You never see all the answers, only all the answers that have shown up so far. Yet we act on what we see. If you see a question 5 minutes in when it only has one answer, is your vote on that answer somehow less valid than mine an hour later when there were three answers? – Monica Cellio Mar 20 '14 at 20:36
• @Shog, sorry, looks like I hadn't read your question properly. So now, any flag we raise will send the post to the 'Low Quality' queue and will need to be reviewed regardless of score. Regardless of score, we will be able to vote to delete (or 'recommend deletion'). That's cool with me, and makes a lot of sense. This actually will really improve the workflow as well (and allow more people to act on bad answers if we educate them). Thanks for the clarification. – jmac Mar 21 '14 at 2:03
• @shog, you explicitly asked for use cases from users of the queue, but it feels like all you're doing is arguing against the ones we've brought up. If I misunderstood your purpose I aoologize. – Monica Cellio Mar 21 '14 at 23:17
• Ok. When I've participated in user studies or other requirements-gathering activities, we've generally gathered a round of feedback and then tried to look at it in the aggregate, rather than dealing with cases one at a time in isolation, but I don't know how common each approach is. C'est la vie. – Monica Cellio Mar 23 '14 at 2:07
• There's a place for that. But honestly, I'm not exactly gathering requirements here - I'm looking for requirements that I overlooked. Flagging is a good one: there are a few different scenarios where that's required - where some systems assume it's possible in fact. – Shog9 Mar 23 '14 at 19:04

You could get rid of a lot of the Not-An-Answer flags by allowing low-rep users to comment. Their inability to comment causes them to create an answer which will ultimately generate a NAA flag.

• But what's your point? This proposal is not about a vast number of NAA flags, it's about the 10k flags queue being pointless and the need to get rid of it. – animuson Mar 20 '14 at 0:54
• @animuson, he's is actually covering the whole flag issue while he's doing it, and specifically a revamp. Fixing this problem should be part of the revamp. – Lance Roberts Mar 20 '14 at 0:55
• @Braiam I have encountered many users who have posted comments as answers. When I approach them, they tell me something like "I know that but I don't have enough rep to comment so I decided to do it this way". – Walker Mar 20 '14 at 0:56
• @Lance That's if you consider it a problem. To be honest, most NAA flags I handle are users asking their own questions in the answers, not users posting comments as answers. Letting low-rep users comment won't solve the problem, because a lot of these answers wouldn't fit in comments anyways. – animuson Mar 20 '14 at 0:57
• You're absolutely correct, Lance. But, then we'd have more "lousy comment" flags, and the comment flag tools ALL suck. – Shog9 Mar 20 '14 at 1:49
• @Shog9, yes, the endless problem of problem fixing. – Lance Roberts Mar 20 '14 at 1:50
• This is a problem I've been chewing on a while Lance, and you're right. My idea is for an incremental privilege comment system - You start out with at most 2 per day, then the more comments you leave that aren't flagged, the more you get, and you lose comments per day as things are flagged. Still hitting the problem of comment moderation tools being awful. I've been wanting to do precisely what you suggest ever since I was a moderator, there's just no easy way to do it. Don't see why this got -13, it's a perfectly logical suggestion. – Tim Post Mar 20 '14 at 3:19
• This could work, but only with @TimPost's proposed idea. +1 – jmort253 Mar 20 '14 at 4:05
• The last thing I want to deal with is spam in the comments... – ɥʇǝS Mar 20 '14 at 4:12
• @TimPost that's a very interesting idea, is it documented somewhere (e.g. feature request) or just here in this fragile comment? – Shadow The Vaccinated Wizard Mar 20 '14 at 7:50
• I like the incremental comment system @TimPost mentions. It rewards the good users, but limits the spammers and trolls. Fits perfectly in the spirit of the SE network. – Toon Krijthe Mar 20 '14 at 8:01
• @Tim Post: The onus on people everywhere to flag comments is going to have to be much, much heavier then. – BoltClock's a Unicorn Mar 20 '14 at 16:08
• @ShadowWizard It's kind of documented in a few places, Jon was doing a lot of research into comments, I've done a lot of thinking about comment moderation, but no coherent spec or feature request yet. As things are going, I might conceivably be able to get to it in 2014, I'd like to, but other things (like the tag wiki system) seem more deserving of love and dev time since they're not ephemeral by nature :) – Tim Post Mar 20 '14 at 16:18
• @seth We already get that to a degree, and the spam protection layer would of course be tied into it, it wouldn't be a big problem. One validated spam flag could take away your ability to comment for a longer period of time, for instance. But, probably not the place to hash it out :) – Tim Post Mar 20 '14 at 16:19
• Fair enough @Tim, and now we also have the new profile page... :) – Shadow The Vaccinated Wizard Mar 20 '14 at 16:23