Let's get rid of the 10K flag queue

Question

The 10K tools are pretty cool... You get a birds-eye view of activity on the site, a "dashboard" view of what's happening. Some of the individual tools haven't scaled particularly well with Stack Overflow's growth, but the concept behind them is still sound: we trust you to enough to be a bee watcher now.

...and then there's the flag queue. What a let-down!

Once upon a time, this queue contained spam and offensive flags, which one might reasonably assume were important enough to put in front of the more trusted members of the site. Re-flagging them brought them that much closer to deletion, while editing them offered salvation to some hapless author.

Nowadays, it's a bunch of Not an Answer flags and a smattering of assorted cruft. 10K users can't even vote to delete these; only 20K users have that privilege. Re-flagging them does nothing but increase their priority in the moderator flag queue, where they frequently outrank more pressing issues; disputing them has its own issues. Also, it's full of bugs, and the behavior has diverged far enough from that of the moderator flag queue on which it is based that it has become an active hindrance to further development of the tooling there.

Worst of all, we have a much better tool for handling crap answers that's available to anyone with the editing privilege. It even has logic built in to prioritize answers likely to be deleted for users with the reputation to delete them. It's like we gave you a car for your 10th birthday, and then replaced it with a rusty bicycle when you turned 16.

It was a nice idea, but it has outlived its usefulness.

Proposed changes

The /tools/flagged route goes away. Period. No replacement. The rest of the 10K tools stick around as informational pages.

Not an Answer flags go into /review/low-quality, just like Very Low Quality flags already do.

Then we beef up the Low Quality review process to make better use of more experienced reviewers and solve this whole "declined / helpful / disputed" flag debate once and for all:

• Effective # of reviews required == ReviewsRequired + # of applicable flags (where ReviewsRequired is 2 on Stack Overflow, 1 everywhere else). So 1 VLQ or NAA flag means EffectiveReviewsRequired=3.
• LQ tasks are not dequeued until one of the following conditions is met:
  1. Post is edited from within review.
     Outcome: flags are marked "helpful" (current behavior).
  2. Post accumulates 3 Delete votes (can only happen when post scores <= 0 and reviewers have >= 20K rep).
     Outcome: post is deleted, flags are marked "helpful".
  3. Task accumulates EffectiveReviewsRequired "Looks Good" reviews.
     Outcome: if the number of (Recommend)Delete reviews is >= the number of Looks Good reviews, mark flags "disputed" and raise DisputedLowQuality mod flag. Otherwise, mark flags "declined".
  4. Task accumulates 6* RecommendDelete + Delete reviews.
     Outcome: mark flags "helpful". If the post scores > 0 then raise DisputedLowQuality mod flag, else just delete post (current behavior).
• As under the current system, flags on posts that've already completed one full review cycle without being deleted should skip /review and go directly into the mod queue.

End result

• Flags get handled faster, more accurately, and with less wasted effort from 10K reviewers.
• Moderators are free to focus on situations that can't be handled by the community - the exceptional cases!
• Developers are free to make enhancements to the moderator tooling without having to work around 10K user restrictions.

Questions

Am I forgetting anything here? I haven't really spent much time in the 10K flag queue since back when it was filled with spam flags and therefore useful - is there a use-case that you depend on that would be lost with its removal? Post 'em below.

*_{: on Stack Overflow, only 4 Recommend Deletion reviews are sufficient.}

Answer 1

This is done:

• /tools/flagged is no more. Disputed flags are still a thing, but disputes are situations observed by the system, not explicit actions. The guidance for new 10K users has been revised: https://stackoverflow.com/help/privileges/moderator-tools

• Not an Answer flags now feed into the Low Quality review queue. This was mostly implemented as described in the proposal above, but since that was written as an implementation guide here's a somewhat more readable summary of the rules:

  1. VLQ and NAA flags immediately enter the mod flag queue in all cases
  2. VLQ and NAA flags on posts that have not been previously reviewed and are not closed, deleted, locked or accepted will enter /review/low-quality-posts.
  3. If review completes successfully, the flag is marked handled and removed from the mod flag queue
  4. If all mod-flags on a post are dismissed from the moderator flag queue, the review task is invalidated
  5. If the outcome of review is deletion, and a post's score prevents it from being deleted, the original flags will be marked "helpful". In these cases, and in cases where the flags are disputed, a new, moderator-only flag will be raised on the post.

We've been operating with these rules in place for about a week now. During this time, 1656 review tasks have been created and processed based on Not An Answer flags:

• 339 of these were resolved as Looks Good
• 1251 of these were resolved as Delete
• 122 "disputed low quality review" flags were raised as a result of conflicting reviews

A couple of observations can be made from this easily:

• 75% of the posts reviewed are considered worthy of deletion by reviewers. Adding in the number of "disputed" tasks puts this at ~83%, which is reasonably close to the 88% accuracy rate on NAA flags handled by moderators in March. More on the discrepancy in a bit.
• This has effectively reduced the "not answer" workload for moderators to less than 10% of what it would have been otherwise.

Now, about that discrepancy... We expected to see this dip a little bit: reviewers as a group are somewhat more tolerant of borderline answers than are moderators. In addition, there are a few other factors at work here:

• Canned comments prompting changes to answers before they finish review
• A backlog of several hundred NAA flags that hadn't been urgent enough to process before now and may have contained a larger % of borderline or straight-up invalid flags.
• As noted above, some of these flags are still handled directly in the moderator queue. 947 review tasks were invalidated due to the flags being handled outside of review, with 908 of the associated posts being deleted. This alone, added to the numbers above, puts the % of helpful flags very close to 88%.

Overall, the results so far have met or exceeded expectations. We'll continue to monitor this - please report any problems you observe.

Answer 2

I've had a request like this on the back-burner for a while. In a nutshell:

• The 10k queue is bad because most of what is there cannot be dealt with by its public.
• Even when 10kers can contribute, they can rarely remove the item from the queue.

As a user on Stack Overflow and elsewhere, I used to participate a bit in the 10k queue back in the old days, but I stopped because I didn't seem to achieve anything there. The only flags that were useful were the close flags and the spam flags. Close flags thankfully moved to the close review queue which is a much saner approach, and spam flags are now shown to mods only (I assume this is because they were abused, which is a pity because fast community intervention would be better than having to wait for a mod to come along). Now, it seems to me that pretty much anything that's left is:

• either things that I can't deal with — I can help by commenting, but so can anyone, and I can help by investigating, but not much;
• or answers that are to be deleted, which requires 20k rep, not 10k.

Feeding close flags into the same close/don't-close queue as close votes was definitely the right thing. Similarly, delete flags (a.k.a. not-an-answer or very-low-quality — but there's no point in having two different flag names for the same effect) should feed into a single delete queue, together with posts with delete votes (thresholds may need to be adjusted accordingly). The point is to show people things that they can actually deal with, in such a way that when enough people act the thing is done without any moderator involvement.

As a moderator — but only on smaller sites, not on SO — I don't remember ever seeing anything useful come out of the 10k flag queue. Pile-on flags are useless: it's my job to review the flags — whether there are 5 NAA flags or 1 NAA flag and 3 “invalid flag” flags, I still have to make my own opinion.

Inasmuch as Community's flags are helpful — and they rarely are — dealing with them typically requires moderator intervention. Consecutive close questions? Check the user's deleted questions, resolve by maybe warning or suspending the user. Possible self-vandalism? Check the user's recent deletions, resolve by maybe warning or suspending the user and undeleting. Excessively long? Why is there even a flag for that? The only one that could be dealt with by non-moderators is “duplicate answers” (possible close as duplicate votes, and deletion of the answer); and this one often requires moderator involvement anyway to delete the answer, if it's been upvoted.

A couple of things I'd add to the delete review (or low quality review if you want to call it that way, but I think delete review is clearer, similar to close review):

• Make sure that there is a clear way to explain why the post should be deleted. Any comment left on the post should notify the user.
• Provide a formal way to indicate when an answer should be converted to a comment. Often flaggers use “not an answer” (as officially recommended), leaving the moderator to figure out again whether the answer should be removed outright or converted to a comment. Even when someone raises a custom flag, often that flag is not as helpful as it could be — flags like “please convert to comment on Bob's answer” (or worse, “please convert to comment on the top answer”…) are an unnecessary communication hindrance.

(Sorry for the length, I don't have time to make this post shorter.)

Answer 3

The 10k flag queue has a lot of problems and I'd love to see it be replaced by a better low-quality-posts queue. There are some use cases that would be negatively affected by the proposed change, so I'd like to see a couple changes to the proposal.

The purpose of this queue (whatever form it takes) is to allow the community to act on poor content, improving it where possible and responding to it in various ways where not. Depending on the nature of the problem, a few different things can happen here:

• Sometimes a post is pure crap and the correct response is to delete it. (10kers can't do that, only request it.)
• Sometimes the reviewer can edit to fix the problem. Great, do that! But we can't assume that everybody who is expert enough to spot a problem is also expert enough to fix it, so "then just edit it" isn't a response that always fits.
• Sometimes it just needs a downvote (and optional comment) -- it's a bad answer but it's an answer and it complies with site guidelines. In this case the flag queue (or review queue) serves to bring bad posts to the attention of higher-rep users who missed them on the front page.
• Sometimes a post might be salvagable, but it requires the author or some other very small subset of users to do it (e.g. if it fails to cite sources but there's some reason to believe sources might exist). These posts should generally be annotated (with comment to ping), with some reminder mechanism so we can check back in a couple days. (Alternatively, I sometimes see these posts get deleted with a comment saying "flag for review once you've fixed it".) Annotations can only be added by moderators at present, so in this case the post needs to be re-flagged for moderator attention. And somebody has to remember to check back later.
• Sometimes the post is just fine. Not all flags have merit.

For all its problems, the 10k flag queue allows reviewers to vote, edit, comment, and flag. I don't spend much time in the low-quality review queue because I can't downvote, I can't flag, and if my reaction is "meh" I have to either say "looks good" when it's not good, just ok, or edit (but what if I personally can't fix it?), or (what I usually do) skip it (which doesn't help move it out of the queue). Processing this queue usually feels like a waste of time.¹

These reviews are particularly important for answers. Answers are the bread and butter (or pearls) of Stack Exchange; that's why we're here. We need effective tools for dealing with poor ones. The 10k queue isn't a good tool, but it provides some capabilities that should be preserved in its replacement. Ideally we make the replacement even better, e.g. by allowing the community to vote on post notices, but let's at least not lose functionality.

People who spend most of their time on sites where answers are testable and are either right or wrong may not see the magnitude of the problem. Some sites suffer the double whammy of (a) being more subjective (answers are hard to verify) and (b) covering topics where everybody thinks he's an expert. Most of us work, so we're obviously qualified to answer any question on The Workplace. Any parent is obviously qualified to answer questions on Parenting. Anybody with some religious background is obviously qualified to answer questions about the bible. Or so the theory goes -- but it's wrong. Sites venturing into these fuzzier, subjective areas need to be able to weed out the bad answers, because they sure do get a lot of them. (I've got dozens more examples where those came from; I took the ones I had handy, which aren't necessarily even the best examples.)

The SE model doesn't work well for sites that attract mostly subjective, unverifiable, opinion-based answers. Yet SE has such sites, and their users can't rely on mods to deal with answer-quality problems (as Shog often says, it's not their job), so there have to be better ways for the community to act. And if not "better", at least "not worse than now".

I like the proposed move of the 10k flags queue into an improved low-quality review queue, so long as this new queue supports voting and flagging. As a stretch goal, please consider incorporating some way for the community to vote on post annotations.

---

¹ I'm aware that I can always leave the review queue (or launch another tab) to do things not possible from the queue. That's a burden, though, and, demonstrably, one I often don't take on. We often hear that people don't downvote enough, in particular; we already have to get past the "oh no my precious 1 rep!" barrier, so let's not add another one for this in particular. Votes should be the primary community response to posts; make it easy.

Answer 4

I'm all for the restructuring of the 10k tools. That used to be a place that I found a lot of value before I was a moderator, but the things that I found of value there (a list of questions with 3+ close votes, access to Late Answers and First Posts, etc.) now have new homes in the review system.

Frankly, the "Not An Answer" list of flags is often being abused now as a means for people to farm helpful flags (which really got bad with the Winterbash hats and around moderator elections), and I don't get a lot of value out of the number of people flagging something as "Not An Answer". When you've seen good answers with 30+ "Not An Answer" flags on them, you take all of that with a grain of salt. One flag is enough.

However, I want to expand on Lance's answer, because I think he brings up a very good point. If we're examining how to improve the workflow behind handling non-answers, I think it helps to look at root causes behind these answers. Ignoring spam and gibberish, almost all non-answers are attempts to comment on something.

New users come here and want to add a little more information to a question or answer and find that they can't. They read the restriction that you need a certain amount of reputation to leave a comment and say "screw that, I'm not finding something to quickly answer just so I can leave a comment here". Many leave and don't come back, but some ignore our warnings and leave their comment as an answer. This is a frustrating experience for a new user who just wants to chip in a little more information, or perhaps correct a minor issue with a question or answer.

The point of the reputation bar that a user has to clear is to prevent comment spam, which would be a very real problem if everyone could comment. However, I believe that if we had better tools to moderate comments in the same way we can questions and answers, it would be possible to watch over and deal with problematic comments from new users.

I've run this past other moderators, but I believe an equivalent for the First Posts and Late Answers review queues could be applied to comments as well. For moderators (and potentially 10k+ users) we could expose a list of recent comments, and make comments searchable to look for patterns of spam. We could add a new category of comment flags for spam and offensive content, weighted in the moderator tools like current spam flags.

With those tools, and with some of the limitations expressed by Tim in his comment on Lance's answer, I believe we could open up commenting to 1-rep registered users and significantly reduce the frequency of non-answers. Yes, you'll see more chatty comments come in ("me too, I'm having this problem", etc.), but the community will be able to flag those for removal just as we can non-answers.

Overall, I think that would make for a much less frustrating experience for newer users visiting the site, hopefully leading to some of them converting into more active members, and reduce the amount of work needed to keep the site clean. It also would give us the means to deal with spammers who somehow pass the current reputation threshold for leaving comments (which has happened, and we had no means of detecting or managing this).

Answer 5

Now days, it's a bunch of Not an Answer flags and a smattering of assorted cruft. 10K users can't even vote to delete these; only 20K users have that privilege. Re-flagging them does nothing but increase their priority in the moderator flag queue, where they frequently outrank more pressing issues

So the issues are:

1. The community can't actually handle the flags there
2. Taking the time to re-flag doesn't actually do anything about the content
3. The entire process screws with moderator priority

Eliminating the queue, while certainly the right thing to do, will not solve the problem by itself, it will just shift the problem to a different queue which may be better suited to deal with the problem, but still won't solve the actual cause.

It's the bad answers, stupid

1. Why can't most users handle the flags? Because the flags are on bad answers.
2. Why doesn't the flagging solve the problem? Because the flags are designed to identify content, not improve it
3. Why does this screw with moderator priority? Because poor answers should be handled by the community, like questions

Questions have a far smoother workflow with 1,657k reviews for 6,909k questions (23.9%), while answers have only 661k reviews for 12,174k answers (5.4%).

Without any good way to deal with poor answers, people raise flags about them (Not an Answer, and Very Low Quality, both which end up in the flag queue). And those flags just end up passing the buck to the moderators. So to fix the problem, we should attack the source.

The tenets of community moderation

There are three types of posts. The community needs a way to distinguish between the three to take the appropriate action (or to allow the mods/system to take appropriate action):

1. Good -> No Moderation Required
2. Needs Improvement -> Give Guidance/Time to Improve
3. Bad -> Remove

Where answers are getting stuck is on that second point. When the close votes were changed last year, here is what the SE team said:

Now, it’s not that we want those questions, but we need to convey exactly why we don’t want them. Imagine if police could give out summons that, rather than, “failure to stop at a signal,” just read, “behavioral violation”. When feedback isn’t specific, it’s impossible to fix the problem, but easy to write it off as probably coming from a bunch of grumpy old jerkfaces who’d rather make you look like an idiot than actually help you.

There is no way for the community to do this save through comments. While we are encouraged to add comments when we downvote, it certainly isn't mandatory (much to the chagrin of the folks being downvoted).

Not to mention that in addition to not explaining how to improve the post, we don't even explain if it can be improved. Some answers should just be removed immediately, such as:

• People posting a question as an answer
• People saying 'I have this problem too' as an answer
• People posting a comment as an answer

For answers that can be improved, there is absolutely no system to see if they have been improved. Again, from the change to close vote reasons:

The goal was always for some closures to drive an edit, improve, re-open cycle. The user gets helped, gets better at asking, and the community gets useful content. Unfortunately, since there was no way to know when a question had been improved, this almost never happened.

The same has been said about answers but has been marked status-declined because we are told that @notifications are good enough if people leave comments.

Dealing with poor answers requires more individual effort than questions

The close system has been around forever. Here's a description of its purpose:

When you vote to close a question, you are really voting for that question to be eventually deleted ... a closed question is no longer alive in any meaningful way, and certainly well on its way to the bit-bucket of /dev/null.

We have since revised that to say, "so long as nobody tries to improve it within 5 days", but the base functionality is the same. Bad content gets closed, and then it will be dealt with by the system (either reviewed and reopened if improved, or auto-deleted if it isn't).

It doesn't work that way for answers. I can downvote, but will never be informed if it's changed unless I check it myself or am @notified. If it isn't improved, it won't be auto-deleted by the system, and unless it collects many downvotes, it won't even be reviewed for deletion by the community. The only exception is if I have over 20k reputation and actively look for answers to delete.

Close votes are free, downvotes aren't (which means people are often reluctant to downvote). Close votes are clear that you think the content isn't good in its current state and gives it a chance to become good, or be discarded. Downvotes are no substitute for close votes, and I don't think anyone would advocate we eliminate the close vote system entirely and rely on downvotes, comments, and @notifications for community moderation of question content.

Allow the community to moderate answers

While I have a separate feature request, I'm not hung up on the details. The fundamental issue is that the community is not being given the tools to deal with poor answers. Since we do not have the tools, we end up creating additional burden for the moderators not because we want to, but because we just don't have anything else we can do.

What would the flag queue look like if the community could do the following?

1. Convert answers to comments
2. Convert questions-as-answers to separate questions
3. Add post notices to answers for improvement

We need a tool to allow us to do with answers what we do with questions. That is what the moderator privileges are all about -- giving us the tools to help moderate more given our rep. Giving us the right tools will allow us to give feedback on answers that allows actionable improvement on the part of askers, and it will allow the system to identify what posts are problematic and need revisiting if edited, or removal if not.

Shifting the Not an Answer/Very Low Quality flags to the Low Quality queue may reduce the moderator burden and increase the ability of the community to handle some of these flags, but it won't solve the fundamental problem: the community has no workflow to deal with poor answers.

---

In a comment, Shog asked for more details on the following statement:

the current tool has only one benefit which it was likely never intended to: when reviewing the flags, I find nice places to deposit my downvotes. This increases the chance that these answers will end up in the VLQ queue and can be deleted. This is not the intended behavior, but it does currently work. So long as downvotes are rare, and upvotes from passersby are common, the VLQ queue will remain sparse and underused.

When we get poor answers, someone will often flag it on TWP. That adds that nice blue moderator tools box to our top bar and increases the eyes of high-rep users on those poor answers. While the flags themselves may not help anything, it does bring our attention (and downvotes) to those answers. If we have enough high-rep users around, this will often push an answer down below zero (quite a rarity on our site), and actually get it tossed in to the VLQ review queue as a result.

We have just recently (past month) started having more active flagging, and it has had a large impact on the size of the VLQ queue. Since the data explorer doesn't actually track the data, I can't give exact numbers, but for reference we have had 531 reviews to date. Today alone we have had 21 reviews, and my guesstimate would be that the amount of NAA/VLQ flags issued correlates pretty well with the number of flags issued on answers.

I do not think this is a good reason to keep the queue around, the result is good but it doesn't mean the system is. But it is something to bear in mind. If this 10k flag queue is eliminated, please watch the more subjective sites to see if there is a dip in downvoting or items in the VLQ review queue after the change.

Answer 6

You could get rid of a lot of the Not-An-Answer flags by allowing low-rep users to comment. Their inability to comment causes them to create an answer which will ultimately generate a NAA flag.