12

There appears to be a bit of a dispute going on in the FAQ answer here. On one hand, a group mostly consisting of Stack Overflow moderators says that if it isn't immediately clear that an answer isn't an answer just by briefly glancing at it, the standard "not an answer" flag shouldn't be raised and a custom moderator flag should be raised instead, clearly explaining why the answer isn't an answer. On the other hand, another group of users say that the standard flag should be raised on all non-answers, regardless of whether they're blatant or not.

I understand where the SO moderators are coming from, because they don't see the question in context or comments when reviewing flags, and they get a lot of flags every day that they need to handle quickly. Thus, it's entirely understandable that they would prefer that the flags they handle contain a clear description of why the answer...isn't.

However, using custom flags places more workload on moderators and bypasses the community review process. The standard "not an answer" flag also places the answer into the Low Quality Posts review queue, where community members review the flagged answers. Additionally, community members reviewing are shown more context than moderators reviewing in their queue. I can think of some other reasons why raising the standard flag even in non-clear-cut cases is better:

  • The standard flag goes immediately to the community "Low Quality Posts" queue, while it is kept out of the moderator queue for a period of time (15 minutes on most sites, 60 minutes on Stack Overflow and some other large sites, no delay on meta sites). This gives time for the community to handle these flags and reduces workload for moderators.

  • If someone casts a standard flag while someone else casts a custom flag, and the answer is deleted in review before a moderator gets around to handling the custom flag, the custom flag isn't dismissed automatically (for good reason) and a moderator still needs to process it.

Based on this, if an answer is a non-answer, but doesn't look like a non-answer at first glance, should I still raise a standard "not an answer" flag, or should I raise a custom flag? An answer from the team would be very nice, though opinions from other users (especially moderators on other sites) are cool too.

3
  • The problem is that the debate is over what it means to be a "blatant non-answer". Jun 1, 2018 at 3:00
  • @curiousdannii: You posted the above comment two minutes before praising Shog9's edit on the original FAQ entry. Is your comment still relevant?
    – user102937
    Jun 4, 2018 at 18:03
  • 1
    @Robert Yes, to my knowledge not everyone agrees that answers to other questions are blatant non answers. Jun 4, 2018 at 23:49

3 Answers 3

17

First of all... This isn't a "Stack Overflow vs. The Network" issue.

Yes, Stack Overflow gets a tremendous number of non-answers, and associated flags, and has to handle them efficiently somehow...

...But so does Super User. So does Ask Ubuntu. So do Web Apps, Electrical Engineering, Arqade...

Any site that has (or appears to have) popular appeal and that gets a lot of traffic from search engines will tend to have this problem: lots of "answers" that are really just responses... And which are blatantly obvious as such.

And yes, all of these sites have that other problem too: the one where someone posts something that looks like an answer, but doesn't actually answer the question at all:

  • link-only answers
  • nonsense bits of other answers mashed together by a spammer trying to get a foot in the door
  • an answer to some other question that the author posted in the wrong place... accidentally?
  • Joke answers
  • "Joke" answers where the joke is that the author intends to claim it was a joke if they get called on it (roll on snare drum)
  • Someone's personal time cube theory which is, of course, the answer to everything (but not a very useful one)
  • The word "NO" in 22pt font w/ a HTML comment to pad out the 30-character minimum answer length
  • et cetera ad nauseam (or other variations on fake latin)

Thing is... These are all a drop in the bucket compared to the first type of problem. And while it may take a moderator a bit of care and research to handle them, they're so rare relative to that first problem as to make them hardly worth spending too much time on in a FAQ designed to help folks learn to flag effectively; unless you make it your mission to hound Contemporary Time Cube Guy and never flag anything else, you're going to spend even less energy flagging this type of thing than the moderators will handling it.

So why does this keep coming up?

Well... I have a theory. Actually, I have two theories, but the first one is mostly just "obsessive meta people" so we can probably skip past that to the more generous second one: lots of people don't know what to do when their flag gets declined.

This is, quite arguably, a fault of the UI. We don't really tell people what to do, other than "learn from it somehow". So, they do - and they come away with really weird ideas of how moderators spend their time handling flags.

So if we're going to fix the FAQ, those misconceptions would be a good place to start. Here's what I've done:

  • Corrected the information about how review factors into this: answers get one trip through review, no matter how many flags they get. If an answer gets 30 NAA flags before it gets reviewed, then ALL of those flags get reviewed.
  • Correct the information about what reviewers can see for context: not much more than what moderators can see.
  • Use the contemporary name for the "custom" flag option.
  • Highlight the two most common reasons for declined, legitimate NAA flags: answers that got fixed, and problems too subtle for the mod or reviewers to notice.
  • Provide explicit advice for escalating after a declined or disputed flag: this was there before, but it was couched in an explanation for why a moderator might've declined it, which is a risky assumption and ignored that review disputes more flags than moderators decline.
10
  • 2
    Has the team ever considered letting people respond to the anonymous rejection of flags with a short message if they think the moderator erroneously invalidated it? I flagged something as VLQ and I got the canned "dont use VLQ for technical inaccuracies or wrong answers", but I didn't use it for either of those reasons and I'd like to let whoever the mod was know that the canned response they used was wrong. I'm not concerned about the flag (I re-flagged as NAA and it was then validated by a mod) as much as I am about moderators using inaccurate (and therefore confusing) canned responses.
    – TylerH
    May 31, 2018 at 19:48
  • "the one where someone posts something that looks like an answer, but doesn't actually answer the question at all" sadly in your revision that last part wasn't codified. In fact, the whole thing reads that maybe you shouldn't use that flag for those those kind of answers, while you suggest an alternative for the subtle case, the in-your-face one isn't addressed.
    – Braiam
    Jun 1, 2018 at 0:17
  • 1
    Well... You can and did, @tyler: you raised another flag. The UI could definitely make this a more obvious path.
    – Shog9
    Jun 1, 2018 at 2:28
  • The in-your-face one is rarely ever an issue, @braiam.
    – Shog9
    Jun 1, 2018 at 2:30
  • Very good edit, thanks @Shog9! Jun 1, 2018 at 3:02
  • @Shog9 Well there's no guarantee the same mod will see the second flag, is there?
    – TylerH
    Jun 1, 2018 at 5:27
  • When I said "in-your-face" I meant those obvious to the reporter since it had context, while the reviewer doesn't have said context to go by.
    – Braiam
    Jun 1, 2018 at 10:04
  • Yeah, I covered that @braiam.
    – Shog9
    Jun 1, 2018 at 14:43
  • 3
    I have a theory about why this keeps coming up as well, and it's rather different from yours. My theory is that this keeps coming up because cases where the flagged post looks like an answer, but clearly isn't an answer when read in the context of the question, represent the majority of incorrectly-declined NAA flags.
    – Mark
    Jan 3, 2019 at 1:50
  • I suspect this varies by site. I think the majority of Not An Answer flags that I have cast (or wanted to cast) have been the type where it looks like an answer to something but when looked at closely it doesn’t answer the specific question at hand.
    – Alex
    Jul 12, 2021 at 23:42
10

I think the answer to this will vary based on the site and the specifics of the answer being flagged.

The "not an answer" flag was designed to provide a fast path to deletion for things that shouldn't be in the answer area. The idea is that this flag should be quick to review by having moderators get into a delete / don't delete mindset, allowing us to burn through dozens of them at a time. It also was not intended as a super-down-vote to remove technically incorrect material, because we don't want moderators to be final arbiters of what is and isn't technically correct on a site.

This has been complicated by the fact that these flags now feed into community review, where normal members can vote to delete or keep posts. Moderators aren't the only ones handling "not an answer" flags, but we still are the only ones dealing with custom flags.

On Stack Overflow, we get something like 800 "not an answer" flags a day. When Jon ran stats on this in 2014, we marked as helpful ~97% of all flags on answers. Most of those are "not an answer" flags. A lot of people like using "not an answer" for answers that have the audacity to compete with theirs or on answers they think are wrong, so most of the declined cases consist of obviously wrong flags. The more controversial flagged cases we're talking about here are a small fraction of the overall flags we get.

From the early days on Stack Overflow, moderators have drawn a pretty hard line at judging technical content. There's no way that we can possibly have the breadth of topics on Stack Overflow represented by the limited number of moderators we have. Outside of the follow-on question, "me too", and "I like turtles" categories of non-answers, it can be very hard for us to judge whether something that reads like an answer is an actual attempt at answering a question unless we're a subject matter expert.

As a result, moderators on Stack Overflow have tended to direct people to use custom flags in cases where it wasn't obvious from the content of the answer alone that it was a non-answer. With the rate at which people flag content they simply don't like as non-answers, combined with technical material that can often be difficult to parse, I really like to have that extra context.

As to whether this inhibits community review, at the time I wrote this answer, review was only able to handle ~35% of all "not an answer" flags. The review queues on Stack Overflow cannot keep up with the incoming flag volume, and moderators are required to handle the rest to prevent things from jamming up. Additionally, upvoted posts that are flagged as "not an answer" (many of the more contentious cases we're talking about here) don't go into community review and are only presented to moderators.

However, all of the above comes from my experience on Stack Overflow. Smaller sites and those that have answers that are easier to interpret by non-experts will handle these flags differently. When you don't have the flag volume and rate of misuse we see on Stack Overflow, you're more likely to have a moderator give you the benefit of the doubt and read through everything around an answer before handling. Community review is also better able to keep up with flag volume, so you may see more of these being handled by your peers.

Even on those sites, there are many cases where it might be helpful to have that little extra context on a borderline non-answer. I particularly like custom flags in cases where someone wrote a technically-oriented comment in an answer that really was a response to another answer. Sometimes it can be difficult to make the connection, and if someone points it out directly I can be sure I didn't miss something.

2
  • 2
    Since you wrote that answer, the number of "Recommend Deletion" reviews needed to delete an answer in the community queue has been reduced to 4 from the network-wide 6 on Stack Overflow only. May 29, 2018 at 22:23
  • 1
    "The "not an answer" flag was designed to provide a fast path to deletion for things that shouldn't be in the answer area" [citation needed] specially the "fast" part, and I did look for it.
    – Braiam
    May 31, 2018 at 23:34
0

To answer the title of your question, in theory... you shouldn't need to.

The reason for that is simple: if isn't immediately obvious why a post isn't an answer, maybe you are just lacking the correct knowledge to recognize as such. This is kind of a point made by Robert Columbia in his first paragraph. It may looks like something that is above the pay grade and you can't understand it, but since all those words sounds like tech stuff, it should be something, right?

There are some telltales, that this answer is just technobabble, but you need to have seen them before to actually identify them, which is why all but "an answer to some other question that the author posted in the wrong place" in Shog's list have such a clear pathway to deletion. Those others have signs that you can then share with others and tell them "this is what you should look for", which takes us to the next part: all those reasons apply to the flag, why this one doesn't!?

This last is so jarring to the flagger since for them the flag should serve as a report: I saw something bad happening, but I don't have the tools to deal with it, so I ask you to take a look. The asymmetry is born from the assumption of the flagger that those kind of posts would be dealt accordingly, when actually they aren't (at least on that site).

Now, and here comes the most baffling thing of the current situation, while we held reviewers in higher standards by presenting them audits, allowing them to filter by tags, concrete guidance of what to look for and complete context without clicks (the answer and the question it was supposed to answer, that should be all the information that you need to being able to tell whenever is a non-answer or not), moderators aren't given such facilities (unless they use the queue like reviewers).

So, while from a results oriented point of view, an other flag has higher chances to be dealt appropriately, non blatant not an answers flags for some moderators are just enough, like you can see in comments from moderators of non-SO sites on the other answer posted to that question and in other discussions around that topic.

6
  • Before submitting this answer I had it put through some word counter and told me that the reader should take around 2-4 min to read the whole thing... yet got a downvote less than a minute after posted. I at least read the whole thing before voting...
    – Braiam
    May 31, 2018 at 23:36
  • 2
    Some people are fast readers. You really can't put the whole trust into a simple estimator. May 31, 2018 at 23:39
  • Also, I guess some people just don't like what you did here. May 31, 2018 at 23:40
  • @SonictheInclusiveHedgehog well, I'm a fast reader. And even then, that version of the answer went uncontested for more than 6 months, so an old grudge on top of that?
    – Braiam
    May 31, 2018 at 23:42
  • Yeah, about that version of the answer going "uncontested"...see here. May 31, 2018 at 23:45
  • @SonictheInclusiveHedgehog still uncontested. The owner didn't "notice", but the post was active and therefore appeared on the homepage for everyone to see, and nobody through anything wrong was going on?
    – Braiam
    Jun 1, 2018 at 0:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .