Proposed improvement to flagging, for answers

Question

Currently the Flag feature, on answers, offers five choices. Three of them request moderator attention:

• Not an answer
• Very low quality
• "Other"

In the spirit of reducing the burden on moderators, and distributing that work a little bit more evenly around the community, I'm proposing a couple of small tweaks to the way answers are flagged.

1. Eliminate Very low quality as an option. Very low quality posts that are, nevertheless, genuine answers do not need moderator attention - they should just be downvoted. (Further justification for this is here and here)

2. Replace Not an answer with four specific options:

   • Should have been a comment - this is not an answer, it is a comment on the question or on another answer. Flagger would then be required to choose which post it should have been a comment on.

   • Should have been an edit - this is not an answer, it is additional information about the question from the original asker. Only appears if the answer is from the same user as the question.

   • Me-too! non-answer - this answer consists of nothing more than "Me too!" or "Did anyone find the answer?"

   • Conversational and irrelevant - this is not an answer, just a random conversation or rant, or meaningless typing that should be deleted

This makes life much easier on the moderators, because now the moderator is not expected to clean up after the bad answer, they merely have to peek at what happened and click "Agree" or "Disagree". If they agree:

• Should have been a comment converts the answer to a comment

• Should have been an edit converts the answer to an edit and it is appended to the question

• Me-too! non-answer deletes the answer and leaves a form-message for the poster explaining that we don't like "Me too", but they should upvote the question if they think it's important or valuable

• Conversational and irrelevant - deletes the answer and leaves a form-message for the poster explaining that we are a Q&A site, etc. etc.

In fact, if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically, without moderator involvement.

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

Thoughts?

Answer 1

In fact, if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically, without moderator involvement.

I still think this is a potentially bad idea. Closing questions without moderator involvement is okay - there are many places where those show up - in 10k tools, in search, even sometimes on the front page. There is peer-review built in.

Answers do not have the same protection. Deleted answers do not show up anywhere except in the question itself and you have to have 10k rep to even see them at all. There isn't the same level of transparency and odds are that a group of users could go off on deleting people's answers. Granted, the ease of this depends on the formula, but even users with high rep and high flag weight can occasionally be malicious.

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

I can't speak for SO, but I don't think this would be a good idea on smaller sites. Are there plans to roll this out network-wide?

Answer 2

Great stuff. As a 10k+ user on SO, I look at a moderator queue with 300 flags and despair. There's nothing I can do to help clear the vast majority of those flags.

I wonder about that last paragraph, though:

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

One of the main benefits of active moderation, in my opinion, is clearing out the truly awful stuff quickly, faster than the community can. I don't care nearly as much about the two-year-old bad grammar as I do about an offensive rant from five minutes ago, that sparks twenty useless comments.

The automatic moderation you talk about can deal with this. But it needs to have a threshold low enough not to let the garbage fester, while not so low as to lead to vigilante deletion. Not easy to do.

EDIT: You might also consider the number of flags a post receives when deciding when to show the flags to moderators. One flag: probably not a problem. Five flags in the span of ten minutes: problem.

Answer 3

I might as well document our current progress.

1. We now suppress the "very low quality" option. It is only available on closed questions or posts with a score less than 1.
2. To eliminate load on Stack Overflow only, we now suppress all the close flags. They are still visible in the 10k queue
3. Close flags are "self curing". If a user casts a close vote on a post with close flags, the close flag is dismissed as valid.
4. If the close flag is disputed by the community, it is always shown in the mod queue
5. We added rich filtering, so you can work on specific flag types
6. SO Only, we suppress VLQ flags from the mod queue for a period of time, in case they are self curing

We are watching the queue at SO and will update this answer with our progress.

Answer 4

if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically

Thank you!! I asked for that feature a long time ago, IMO it should have been there from the start. Currently flag weight can get you a shiny badge, but once you have that badge it does nothing else for you.

I do think you need one more NAA option: Should have been a new question. There are a small number of NAA flags for this sort of thing, often there is nothing wrong with the question, it's just posted in the wrong place. This should be a simple thing to implement.

Should have been a comment converts the answer to a comment

From what I have seen in the flagged queue, this will not work. You cannot predict which answer it was supposed to be a comment for, especially if the order of the answers has been randomised. We can afford to lose comments, these ones in particular very seldom add value, so just blow it away because it isn't worth saving.

Answer 5

I like the idea of suggesting hints for moderators. There are several users on SO that already do this (some of them are listed here, not coincidentally), and it really speeds up flag processing. It makes sense to encourage it as much as possible on the flag dialog.

In fact, if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically, without moderator involvement.

I'd be okay with this if the formula was based on past data. I've seen posts flagged a dozen times for the same reason, and I don't think I've ever disagreed in those cases. I'm sure there's enough historical data to find a threshold flag weight value that we can be confident in. Naturally, people disputing the flag should either subtract from the total flag weight of the post, or eliminate the post from automatic moderation altogether.

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

Maybe we could partition the moderation queue so that flags that could potentially cure themselves still go into the 10K moderator queue, but don't show up in the diamond moderator queue until they hit a certain flag weight score (or base it on elapsed time as you suggested, or both). Flags that can only be dealt with by using diamond moderator tools (e.g., should be a comment) should not be delayed.

Answer 6

"Me-too! non-answer" could be merged with "Conversational and irrelevant," as any "Me too!" answer is conversational and irrelevant; such answers should not even converted to comments.
The proposed form-message for the "me-too! non-aswer" could still be shown for "conversational and irrelevant" to explain that there are some posts that should not be posted on an SE site.

I would also add "it's a completely different question"; it still common that users write as answer something that should be posted as new question. The difference with "conversational and irrelevant" is that the answer could be converted in a question, or the OP could get a message about how to ask a new question.

Answer 7

Many times, the users on Electrical Engineering post 'other' flags like your examples, especially 'should have been a comment' and 'should have been an edit'.

Have you analyzed the previos 'other' flag text and comments left by mods after 'not an answer' flags to determine what the most frequent problems are?

One discrepancy with my experience is that

• Conversational and irrelevant - this is not an answer, just a random conversation or rant, or meaningless typing that should be deleted

appears to be applicable to a subset of "this isn't a forum or mailing list" type posts. As often as I see conversational and irrelevant continued discussions, I see fairly legitimate questions posted as answers. Also, 'that should be deleted' might seem a little strong to prospective flaggers. Let the mods decide what should be done with it.

Perhaps

• Forum post - this is not an answer, just a random conversation, rant, or a new question

would be better? Or even reusing the good ol' not an answer flag reason with some new text?

Answer 8

As noted in my own very similar question last week, I would favour allocating flags by recommended action. It is not necessary to distinguish between a "me too" post and pure noise: they both just need to be deleted. I agree that "convert to comment" should be an option - this is one of the things I proposed last week.

I also think we need to have a choice of "Recommend deleting user account". Recently I have been using the "Other" flag for this, as in this message I just pulled out of my flagging summary.

Five posts from the same new user, all identical, all linking to the same low-quality "development resource" web site. Please delete this user's account

Of course, I could have just flagged all five posts (I am that close to the Marshal badge for SO as it is), but this approach seems to be more efficient for moderators.

I don't favour leaving a message in place of "me too" posts: most of the users that leave them don't have enough reputation to upvote or comment. That's why they post non-answers as answers.

Answer 9

I like having more granularity, as long as there's always a generic flag.

I don't agree with getting rid of the 'Very Low Quality' flag. There are times it needs to be used (though not a huge amount), but we don't want those question/answers hanging around polluting up the site.

Here's one example, not the most egregious type of example, but do we really want to let this hang out on SO by only downvoting it?

Answer 10

I don't have much feedback on the question, I very rarely flag answers. When I see a bad answer I either post a comment to help the poster to improve his answer, downvote it when it is utter crap or post a better answer. Asking a moderator to do this for me makes little sense to me.

I do not envy their job working down the queue though. I won't mind helping out but that's gotten very difficult in the past year and a half. The SO 10K moderator page has gotten too hard to handle. That starts with the creepy number displayed on the toolbar, easily climbing to hundreds on a weekend. Ends with the actual page that crams 50 posts in dense text, often referring to tags that I have no interest in.

Please consider filtering this page:

• select only tags that I have favorited, same idea as the front page
• give me a manageable chunk to give me a feeling I finished something. Say 15.
• don't show questions I already looked at and voted for.

Answer 11

I don't think too much weight should be put on flag weight if you decide to go with automatic actions. On SO, anyone with 15+ rep can get 500 flag weight in just 3 days and 600 in another day. And even reaching 750 would only take 2-3 weeks.

Also, if anyone flags it as an invalid flag I'd suggest that it should not be eligible for automatic handling and it should instead be dealt with by a moderator.

Answer 12

1. The moderator queue tends to fill up with a lot of things that would have cured themselves if left alone for a few minutes.
2. Questions that are marked as low quality which are often edited into better quality by the time a mod looks at the queue, for example. Ultimately the goal is to use moderators as a last resort where the community fails to do things, because otherwise they end up doing too much work (and might burn out).

Its not fair enough that the mod queue has become a way to push the burden of managing huge quantity of questions a day with a small bunch of working personalities.