Should a custom flag be used instead of the "not an answer" flag when the answer isn't a blatant non-answer?

Question

There appears to be a bit of a dispute going on in the FAQ answer here. On one hand, a group mostly consisting of Stack Overflow moderators says that if it isn't immediately clear that an answer isn't an answer just by briefly glancing at it, the standard "not an answer" flag shouldn't be raised and a custom moderator flag should be raised instead, clearly explaining why the answer isn't an answer. On the other hand, another group of users say that the standard flag should be raised on all non-answers, regardless of whether they're blatant or not.

I understand where the SO moderators are coming from, because they don't see the question in context or comments when reviewing flags, and they get a lot of flags every day that they need to handle quickly. Thus, it's entirely understandable that they would prefer that the flags they handle contain a clear description of why the answer...isn't.

However, using custom flags places more workload on moderators and bypasses the community review process. The standard "not an answer" flag also places the answer into the Low Quality Posts review queue, where community members review the flagged answers. Additionally, community members reviewing are shown more context than moderators reviewing in their queue. I can think of some other reasons why raising the standard flag even in non-clear-cut cases is better:

• The standard flag goes immediately to the community "Low Quality Posts" queue, while it is kept out of the moderator queue for a period of time (15 minutes on most sites, 60 minutes on Stack Overflow and some other large sites, no delay on meta sites). This gives time for the community to handle these flags and reduces workload for moderators.

• If someone casts a standard flag while someone else casts a custom flag, and the answer is deleted in review before a moderator gets around to handling the custom flag, the custom flag isn't dismissed automatically (for good reason) and a moderator still needs to process it.

Based on this, if an answer is a non-answer, but doesn't look like a non-answer at first glance, should I still raise a standard "not an answer" flag, or should I raise a custom flag? An answer from the team would be very nice, though opinions from other users (especially moderators on other sites) are cool too.

Answer 1

First of all... This isn't a "Stack Overflow vs. The Network" issue.

Yes, Stack Overflow gets a tremendous number of non-answers, and associated flags, and has to handle them efficiently somehow...

...But so does Super User. So does Ask Ubuntu. So do Web Apps, Electrical Engineering, Arqade...

Any site that has (or appears to have) popular appeal and that gets a lot of traffic from search engines will tend to have this problem: lots of "answers" that are really just responses... And which are blatantly obvious as such.

And yes, all of these sites have that other problem too: the one where someone posts something that looks like an answer, but doesn't actually answer the question at all:

• link-only answers
• nonsense bits of other answers mashed together by a spammer trying to get a foot in the door
• an answer to some other question that the author posted in the wrong place... accidentally?
• Joke answers
• "Joke" answers where the joke is that the author intends to claim it was a joke if they get called on it (roll on snare drum)
• Someone's personal time cube theory which is, of course, the answer to everything (but not a very useful one)
• The word "NO" in 22pt font w/ a HTML comment to pad out the 30-character minimum answer length
• et cetera ad nauseam (or other variations on fake latin)

Thing is... These are all a drop in the bucket compared to the first type of problem. And while it may take a moderator a bit of care and research to handle them, they're so rare relative to that first problem as to make them hardly worth spending too much time on in a FAQ designed to help folks learn to flag effectively; unless you make it your mission to hound Contemporary Time Cube Guy and never flag anything else, you're going to spend even less energy flagging this type of thing than the moderators will handling it.

So why does this keep coming up?

Well... I have a theory. Actually, I have two theories, but the first one is mostly just "obsessive meta people" so we can probably skip past that to the more generous second one: lots of people don't know what to do when their flag gets declined.

This is, quite arguably, a fault of the UI. We don't really tell people what to do, other than "learn from it somehow". So, they do - and they come away with really weird ideas of how moderators spend their time handling flags.

So if we're going to fix the FAQ, those misconceptions would be a good place to start. Here's what I've done:

• Corrected the information about how review factors into this: answers get one trip through review, no matter how many flags they get. If an answer gets 30 NAA flags before it gets reviewed, then ALL of those flags get reviewed.
• Correct the information about what reviewers can see for context: not much more than what moderators can see.
• Use the contemporary name for the "custom" flag option.
• Highlight the two most common reasons for declined, legitimate NAA flags: answers that got fixed, and problems too subtle for the mod or reviewers to notice.
• Provide explicit advice for escalating after a declined or disputed flag: this was there before, but it was couched in an explanation for why a moderator might've declined it, which is a risky assumption and ignored that review disputes more flags than moderators decline.

Answer 2

I think the answer to this will vary based on the site and the specifics of the answer being flagged.

The "not an answer" flag was designed to provide a fast path to deletion for things that shouldn't be in the answer area. The idea is that this flag should be quick to review by having moderators get into a delete / don't delete mindset, allowing us to burn through dozens of them at a time. It also was not intended as a super-down-vote to remove technically incorrect material, because we don't want moderators to be final arbiters of what is and isn't technically correct on a site.

This has been complicated by the fact that these flags now feed into community review, where normal members can vote to delete or keep posts. Moderators aren't the only ones handling "not an answer" flags, but we still are the only ones dealing with custom flags.

On Stack Overflow, we get something like 800 "not an answer" flags a day. When Jon ran stats on this in 2014, we marked as helpful ~97% of all flags on answers. Most of those are "not an answer" flags. A lot of people like using "not an answer" for answers that have the audacity to compete with theirs or on answers they think are wrong, so most of the declined cases consist of obviously wrong flags. The more controversial flagged cases we're talking about here are a small fraction of the overall flags we get.

From the early days on Stack Overflow, moderators have drawn a pretty hard line at judging technical content. There's no way that we can possibly have the breadth of topics on Stack Overflow represented by the limited number of moderators we have. Outside of the follow-on question, "me too", and "I like turtles" categories of non-answers, it can be very hard for us to judge whether something that reads like an answer is an actual attempt at answering a question unless we're a subject matter expert.

As a result, moderators on Stack Overflow have tended to direct people to use custom flags in cases where it wasn't obvious from the content of the answer alone that it was a non-answer. With the rate at which people flag content they simply don't like as non-answers, combined with technical material that can often be difficult to parse, I really like to have that extra context.

As to whether this inhibits community review, at the time I wrote this answer, review was only able to handle ~35% of all "not an answer" flags. The review queues on Stack Overflow cannot keep up with the incoming flag volume, and moderators are required to handle the rest to prevent things from jamming up. Additionally, upvoted posts that are flagged as "not an answer" (many of the more contentious cases we're talking about here) don't go into community review and are only presented to moderators.

However, all of the above comes from my experience on Stack Overflow. Smaller sites and those that have answers that are easier to interpret by non-experts will handle these flags differently. When you don't have the flag volume and rate of misuse we see on Stack Overflow, you're more likely to have a moderator give you the benefit of the doubt and read through everything around an answer before handling. Community review is also better able to keep up with flag volume, so you may see more of these being handled by your peers.

Even on those sites, there are many cases where it might be helpful to have that little extra context on a borderline non-answer. I particularly like custom flags in cases where someone wrote a technically-oriented comment in an answer that really was a response to another answer. Sometimes it can be difficult to make the connection, and if someone points it out directly I can be sure I didn't miss something.

Answer 3

To answer the title of your question, in theory... you shouldn't need to.

The reason for that is simple: if isn't immediately obvious why a post isn't an answer, maybe you are just lacking the correct knowledge to recognize as such. This is kind of a point made by Robert Columbia in his first paragraph. It may looks like something that is above the pay grade and you can't understand it, but since all those words sounds like tech stuff, it should be something, right?

There are some telltales, that this answer is just technobabble, but you need to have seen them before to actually identify them, which is why all but "an answer to some other question that the author posted in the wrong place" in Shog's list have such a clear pathway to deletion. Those others have signs that you can then share with others and tell them "this is what you should look for", which takes us to the next part: all those reasons apply to the flag, why this one doesn't!?

This last is so jarring to the flagger since for them the flag should serve as a report: I saw something bad happening, but I don't have the tools to deal with it, so I ask you to take a look. The asymmetry is born from the assumption of the flagger that those kind of posts would be dealt accordingly, when actually they aren't (at least on that site).

Now, and here comes the most baffling thing of the current situation, while we held reviewers in higher standards by presenting them audits, allowing them to filter by tags, concrete guidance of what to look for and complete context without clicks (the answer and the question it was supposed to answer, that should be all the information that you need to being able to tell whenever is a non-answer or not), moderators aren't given such facilities (unless they use the queue like reviewers).

So, while from a results oriented point of view, an other flag has higher chances to be dealt appropriately, non blatant not an answers flags for some moderators are just enough, like you can see in comments from moderators of non-SO sites on the other answer posted to that question and in other discussions around that topic.