What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

Currently the Flag feature, on answers, offers five choices. Three of them request moderator attention:

  • Not an answer
  • Very low quality
  • "Other"

In the spirit of reducing the burden on moderators, and distributing that work a little bit more evenly around the community, I'm proposing a couple of small tweaks to the way answers are flagged.

  1. Eliminate Very low quality as an option. Very low quality posts that are, nevertheless, genuine answers do not need moderator attention - they should just be downvoted. (Further justification for this is here and here)

  2. Replace Not an answer with four specific options:

    • Should have been a comment - this is not an answer, it is a comment on the question or on another answer. Flagger would then be required to choose which post it should have been a comment on.

    • Should have been an edit - this is not an answer, it is additional information about the question from the original asker. Only appears if the answer is from the same user as the question.

    • Me-too! non-answer - this answer consists of nothing more than "Me too!" or "Did anyone find the answer?"

    • Conversational and irrelevant - this is not an answer, just a random conversation or rant, or meaningless typing that should be deleted

This makes life much easier on the moderators, because now the moderator is not expected to clean up after the bad answer, they merely have to peek at what happened and click "Agree" or "Disagree". If they agree:

  • Should have been a comment converts the answer to a comment

  • Should have been an edit converts the answer to an edit and it is appended to the question

  • Me-too! non-answer deletes the answer and leaves a form-message for the poster explaining that we don't like "Me too", but they should upvote the question if they think it's important or valuable

  • Conversational and irrelevant - deletes the answer and leaves a form-message for the poster explaining that we are a Q&A site, etc. etc.

In fact, if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically, without moderator involvement.

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

Thoughts?

share|improve this question
1  
I agree - mods have plenty to do, and mitigating a good deal of the work onto 10k+ users who want it is a good solution. I think it should be clarified that "Other" is a perfectly legitimate way to clarify flag reasoning, for moderator and non-moderator flags, though, since I've often in the past been iffy on using them due to the connotation that "Other" is necessarily not at all relevant to the other flag reasons. –  Nightfirecat Oct 27 '11 at 1:45
11  
I like the new roster of flag choices, but I don't know about skipping the flag list or automatic flag application. I'd rather see how the community uses the new flags first before following them blindly. –  badp Oct 27 '11 at 1:48
2  
@badp of course. That's the "FORMULA TBD" part. That said, if three users with very high flag weight all mark an answer as "Me-Too!" it's probably pretty safe to delete. The main point is that because each flag incorporates full information about the desired outcome, it's POSSIBLE for the first time to process them automatically, just like we do with comment flags –  Joel Spolsky Oct 27 '11 at 1:51
7  
I agree with the removal of "very low quality" as a moderator flag, but it would still be useful IMO to capture that information to an area where any user so inclined can go and see a list of posts that need to be edited and edit them. –  Kyle Cronin Oct 27 '11 at 2:24
5  
Possibly related question: meta.stackexchange.com/questions/109886/… Based on my own experience, one of the recommended actions needs to be "delete the account" - for repeat spammers. –  Verbeia Oct 27 '11 at 2:57
    
@Verbeia sorry I missed that! Yes, I guess this is not the world's most original idea. But I hope we can implement it and that it helps. –  Joel Spolsky Oct 27 '11 at 3:02
2  
@Kyle, some questions are of such low quality, there's no way to know how to edit them. –  Lance Roberts Oct 27 '11 at 3:15
1  
@JoelSpolsky - no worries (and I think we have a nice clean test of the upvoting-users-not-questions effect). The "convert to comment" flag has been requested before me, too: meta.stackexchange.com/questions/40906/… –  Verbeia Oct 27 '11 at 4:21
1  
Begin joke: "Should have been a comment" on the five recent questions asking for this. –  Rosinante Oct 27 '11 at 10:19
    
I think anyone with a good flag weight should be able to tick a box to say "look at this NOW", but lose more flag weight if they are wronge. –  Ian Ringrose Oct 27 '11 at 12:21
1  
Delaying putting a flag into the moderator queue, should only be done for large sites, as "moderation happens automatically, without moderator involvement" is unlicly to work for a small site. –  Ian Ringrose Oct 27 '11 at 12:22
3  
Here are my thoughts. –  Won't Oct 27 '11 at 14:47
    
@Verbeia Joel also dropped a trailer for this idea in the podcast... so pretty much everyone knew it was going to be on meta within the next couple of hours after that ;) –  Lorem Ipsum Oct 27 '11 at 23:04
    
What about a "Nothing but a link" option? This is a recognisable problem of the same frequency as 2.a and 2.c. Often these are just converted to comments when flagged as NARA but if we're going to be specific then this should benefit. The auto-action could even be converting to a comment. –  Grant Thomas Nov 3 '11 at 9:23
1  
Please implement this! There are so many answers being flagged with "not an answer" because they are incorrect answers, not because they are unfit for the site. The description "not an answer" is such poor wording that it easily encourages this misuse. Of course, the downvote reputation loss doesn't help either. –  Jacob Jan 19 at 23:21
add comment

12 Answers

In fact, if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically, without moderator involvement.

I still think this is a potentially bad idea. Closing questions without moderator involvement is okay - there are many places where those show up - in 10k tools, in search, even sometimes on the front page. There is peer-review built in.

Answers do not have the same protection. Deleted answers do not show up anywhere except in the question itself and you have to have 10k rep to even see them at all. There isn't the same level of transparency and odds are that a group of users could go off on deleting people's answers. Granted, the ease of this depends on the formula, but even users with high rep and high flag weight can occasionally be malicious.

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

I can't speak for SO, but I don't think this would be a good idea on smaller sites. Are there plans to roll this out network-wide?

share|improve this answer
    
I see the point you are making, but still ask: what is the point of having 700+ flag weight if I can't actually do anything with it? Maybe the solution is to go ahead with the planned change, but have a way for the mods to get a list of the deleted answers so they can review them? The scenario of one high rep/weight user going rogue is possible, but they would quickly be brought into line by the flag weight penalties if they operate alone (it wouldn't be too hard to write a query to detect this behaviour), and I think the chances of a rogue cell of operatives would be pretty low? –  slugster Oct 27 '11 at 21:28
    
@slugster Flag weight was only ever meant to sort flags in the moderator queue. If we start assigning extra meaning and privileges to go with it, I think we should be careful. To your specific suggestion, if moderators have another list to check and review, have we really relieved them of any work? They might as well just review the flag(s) in the first place. –  Anna Lear Oct 27 '11 at 21:40
    
I'm just thinking that your main concern is accountability, so the review feature would be there in case it was needed (i.e. not for regular hour to hour use), I'm all for the appropriate safeguards to be put in place to protect the system. I'm also keen to see this implemented because it can have a quite beneficial impact on the mod queue. –  slugster Oct 27 '11 at 23:37
    
@slugster If the review list doesn't have to be looked at, it won't be. Mods will still be busy. I think if we're going to add accountability measures, they should at least be accessible by more than just moderators. Still, I'd prefer starting with more specific flags and seeing how that does and whether introducing a major change in how answer flags work (deleting something is a big deal) is really needed at all. –  Anna Lear Oct 28 '11 at 0:23
    
"Deleted answers do not show up anywhere except in the question itself" - that's not completely true. The second-class moderator (10k) tools have a list of recently deleted posts that goes back up to 30 days. –  lunboks Jan 14 '12 at 20:26
add comment

Great stuff. As a 10k+ user on SO, I look at a moderator queue with 300 flags and despair. There's nothing I can do to help clear the vast majority of those flags.

I wonder about that last paragraph, though:

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

One of the main benefits of active moderation, in my opinion, is clearing out the truly awful stuff quickly, faster than the community can. I don't care nearly as much about the two-year-old bad grammar as I do about an offensive rant from five minutes ago, that sparks twenty useless comments.

The automatic moderation you talk about can deal with this. But it needs to have a threshold low enough not to let the garbage fester, while not so low as to lead to vigilante deletion. Not easy to do.

EDIT: You might also consider the number of flags a post receives when deciding when to show the flags to moderators. One flag: probably not a problem. Five flags in the span of ten minutes: problem.

share|improve this answer
2  
In the last few days, I've noticed that the moderator queue tends to fill up with a lot of things that would have cured themselves if left alone for a few minutes. Questions that are marked as low quality which are often edited into better quality by the time a mod looks at the queue, for example. Ultimately the goal is to use moderators as a last resort where the community fails to do things, because otherwise they end up doing too much work (and might burn out). I don't like the way that the mod queue has become a way to push the burden of managing 4000 questions a day onto a few shoulders –  Joel Spolsky Oct 27 '11 at 1:43
14  
Fair point, @Joel. On a slightly different note, I've sometimes wished (but haven't asked) for the ability to grade my flags. Or something. I see stuff that should be flagged, I guess, but really isn't a high priority. Then I see stuff that needs to be nuked right now. I have high flag weight, and it seems silly to put the former right on top of the moderator queue (assuming it still works like that). –  Michael Petrotta Oct 27 '11 at 1:50
    
@MichaelPetrotta Would be nice to be able to have a "reset on edit" flag. While an edit (especially by a 3rd party) might not fix all the issues, they often do. –  Richard Nov 1 '11 at 7:41
add comment

I might as well document our current progress.

  1. We now suppress the "very low quality" option. It is only available on closed questions or posts with a score less than 1.
  2. To eliminate load on Stack Overflow only, we now suppress all the close flags. They are still visible in the 10k queue
  3. Close flags are "self curing". If a user casts a close vote on a post with close flags, the close flag is dismissed as valid.
  4. If the close flag is disputed by the community, it is always shown in the mod queue
  5. We added rich filtering, so you can work on specific flag types
  6. SO Only, we suppress VLQ flags from the mod queue for a period of time, in case they are self curing

We are watching the queue at SO and will update this answer with our progress.

share|improve this answer
    
I thought these proposals were only for answer flags. I use VLQ semi-frequently on questions that I think are crappy NARQs and don't seem to be reaching "close velocity". Is the new preferred procedure to use the "other" box (which looks strange on its own, BTW) all the time? –  Josh Caswell Nov 1 '11 at 5:33
    
If you are trying to escalate closure, you are going to have to leave a comment, simply adding a "vlq" just makes us confused. (I will suppress that other radio when it is alone) .... at the moment it looks like answer flags are not much of a problem ... the most difficult thing in the queue now is all the custom stuff –  waffles Nov 1 '11 at 5:38
    
Okey-dokey. Guess I was using them wrong. –  Josh Caswell Nov 1 '11 at 5:42
1  
@JeffAtwood Did you just bump up the threshold to 1, or was it always 1 and waffles posted it wrong? –  NullUserException อ_อ Dec 9 '11 at 7:21
    
@pro I believe very strongly -- violently, even -- that posts with a score of zero should be eligible for VLQ flags. I agree that posts with a score of 1 or higher shouldn't be. That change has been in for a while now. –  Jeff Atwood Dec 9 '11 at 7:37
    
What happens to flag weight under point 3? If I flag a post, and also vote to close it (presumably this is referring to questions, not answers) I assume there is no increase in flag weight, but if I simply flag and wait for someone else to close vote, what happens? –  jrturton Dec 9 '11 at 15:11
    
Someone just referred me to this as evidence of how the current system works, specifically point number 3. Am I correct in interpreting that as saying that if I cast a single close vote (not the final) on any question in the flag queue, all pending close flags will be dismissed as valid? Does this also remove it from the queue entirely, even if it has not been closed yet? –  Cody Gray Mar 20 '12 at 17:27
    
@waffles Are there any plans to implement points #2, and #6 in other sites too? They could be helpful in other sites too. –  kiamlaluno Mar 22 '12 at 14:41
add comment

if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically

Thank you!! I asked for that feature a long time ago, IMO it should have been there from the start. Currently flag weight can get you a shiny badge, but once you have that badge it does nothing else for you.

I do think you need one more NAA option: Should have been a new question. There are a small number of NAA flags for this sort of thing, often there is nothing wrong with the question, it's just posted in the wrong place. This should be a simple thing to implement.

Should have been a comment converts the answer to a comment

From what I have seen in the flagged queue, this will not work. You cannot predict which answer it was supposed to be a comment for, especially if the order of the answers has been randomised. We can afford to lose comments, these ones in particular very seldom add value, so just blow it away because it isn't worth saving.

share|improve this answer
1  
I agree on the "Should have been a new question" reason, but I don't think we want those to be automatically converted to questions, as explained by Jeff here: meta.stackexchange.com/questions/53096/… . However, leaving an auto-comment that directs the user on how to ask a new question (as seems to be done now) would be a good thing. –  Brad Larson Oct 27 '11 at 17:40
    
the more flag weight you have, the higher your flags show up in the queue, so "high flag weight does nothing for you" is not true. If someone with a high flag weight flags, it's prioritized in the list, etc. –  Jeff Atwood Dec 9 '11 at 7:02
    
Thanks @Jeff, good to know that flag weight is actually useful and not just a number. –  slugster Dec 9 '11 at 13:47
add comment

I like the idea of suggesting hints for moderators. There are several users on SO that already do this (some of them are listed here, not coincidentally), and it really speeds up flag processing. It makes sense to encourage it as much as possible on the flag dialog.

In fact, if enough people with high enough flag weight or reputation (FORMULA TBD) apply the same flag, the moderation happens automatically, without moderator involvement.

I'd be okay with this if the formula was based on past data. I've seen posts flagged a dozen times for the same reason, and I don't think I've ever disagreed in those cases. I'm sure there's enough historical data to find a threshold flag weight value that we can be confident in. Naturally, people disputing the flag should either subtract from the total flag weight of the post, or eliminate the post from automatic moderation altogether.

To prevent the moderation queue from filling up with problems that might cure themselves, these flags don't even go in the moderator queue until some period of time (TBD) has passed.

Maybe we could partition the moderation queue so that flags that could potentially cure themselves still go into the 10K moderator queue, but don't show up in the diamond moderator queue until they hit a certain flag weight score (or base it on elapsed time as you suggested, or both). Flags that can only be dealt with by using diamond moderator tools (e.g., should be a comment) should not be delayed.

share|improve this answer
    
Can you give more information about the types of moderator hint you'd like to see? I'd like my flagging to be as helpful as possible, so any advice is welcome. –  jrturton Oct 27 '11 at 17:20
    
@jrturton The best hint you can give us is what specific action you think a moderator should take, like delete, convert to comment (and on which post), or convert to edit. It's also a big help when you tell us what you've already done. For example, it saves us some time if we know that you've already edited a question to include the contents of an "answer." –  Bill the Lizard Oct 27 '11 at 18:14
1  
@BilltheLizard status-completed ... the mod queue now suppresses flag to close stuff. you can still see it in the 10k queue –  waffles Nov 1 '11 at 5:01
add comment

"Me-too! non-answer" could be merged with "Conversational and irrelevant," as any "Me too!" answer is conversational and irrelevant; such answers should not even converted to comments.
The proposed form-message for the "me-too! non-aswer" could still be shown for "conversational and irrelevant" to explain that there are some posts that should not be posted on an SE site.

I would also add "it's a completely different question"; it still common that users write as answer something that should be posted as new question. The difference with "conversational and irrelevant" is that the answer could be converted in a question, or the OP could get a message about how to ask a new question.

share|improve this answer
2  
Merging "Me too!" would remove the feature "leaves a form-message for the poster explaining that we don't like "Me too", but they should upvote the question if they think it's important or valuable." I agree that "it's a completely different question" is common. I suggested that it be merged with "conversational and irrelevant" in my answer. –  Kevin Vermeer Oct 27 '11 at 12:38
add comment

Many times, the users on Electrical Engineering post 'other' flags like your examples, especially 'should have been a comment' and 'should have been an edit'.

Have you analyzed the previos 'other' flag text and comments left by mods after 'not an answer' flags to determine what the most frequent problems are?

One discrepancy with my experience is that

  • Conversational and irrelevant - this is not an answer, just a random conversation or rant, or meaningless typing that should be deleted

appears to be applicable to a subset of "this isn't a forum or mailing list" type posts. As often as I see conversational and irrelevant continued discussions, I see fairly legitimate questions posted as answers. Also, 'that should be deleted' might seem a little strong to prospective flaggers. Let the mods decide what should be done with it.

Perhaps

  • Forum post - this is not an answer, just a random conversation, rant, or a new question

would be better? Or even reusing the good ol' not an answer flag reason with some new text?

share|improve this answer
add comment

As noted in my own very similar question last week, I would favour allocating flags by recommended action. It is not necessary to distinguish between a "me too" post and pure noise: they both just need to be deleted. I agree that "convert to comment" should be an option - this is one of the things I proposed last week.

I also think we need to have a choice of "Recommend deleting user account". Recently I have been using the "Other" flag for this, as in this message I just pulled out of my flagging summary.

Five posts from the same new user, all identical, all linking to the same low-quality "development resource" web site. Please delete this user's account

Of course, I could have just flagged all five posts (I am that close to the Marshal badge for SO as it is), but this approach seems to be more efficient for moderators.

I don't favour leaving a message in place of "me too" posts: most of the users that leave them don't have enough reputation to upvote or comment. That's why they post non-answers as answers.

share|improve this answer
add comment

I like having more granularity, as long as there's always a generic flag.

I don't agree with getting rid of the 'Very Low Quality' flag. There are times it needs to be used (though not a huge amount), but we don't want those question/answers hanging around polluting up the site.

Here's one example, not the most egregious type of example, but do we really want to let this hang out on SO by only downvoting it?

VLQ example

share|improve this answer
12  
That's more of a "not an answer" (should've been, at best, a comment) than "very low quality". The latter should be used for unreadable posts, even though pretty much nobody ever uses it that way.. which is in itself an argument for removing it at least from answers. –  Anna Lear Oct 27 '11 at 5:18
2  
You could also arguably classify a post like that as Conversational and irrelevant. –  Chris Frederick Oct 27 '11 at 5:56
add comment

I don't have much feedback on the question, I very rarely flag answers. When I see a bad answer I either post a comment to help the poster to improve his answer, downvote it when it is utter crap or post a better answer. Asking a moderator to do this for me makes little sense to me.

I do not envy their job working down the queue though. I won't mind helping out but that's gotten very difficult in the past year and a half. The SO 10K moderator page has gotten too hard to handle. That starts with the creepy number displayed on the toolbar, easily climbing to hundreds on a weekend. Ends with the actual page that crams 50 posts in dense text, often referring to tags that I have no interest in.

Please consider filtering this page:

  • select only tags that I have favorited, same idea as the front page
  • give me a manageable chunk to give me a feeling I finished something. Say 15.
  • don't show questions I already looked at and voted for.
share|improve this answer
    
> Asking a moderator to do this for me makes little sense to me. – that's not the point. Moderators should do the stuff you can't do, like deleting or migrating as a comment. –  slhck Oct 27 '11 at 12:51
3  
@slhck "Asking a moderator to do this for me makes little sense to me." I think that's right on point. We do get a lot of moderator flags for things people could just fix themselves. Removing those from the queue will free us up to do the things that everyone else can't. –  Bill the Lizard Oct 27 '11 at 13:03
    
@Bill Yes, I just said that post a comment to help the poster to improve his answer, downvote it when it is utter crap or post a better answer is definitely not something you would do. –  slhck Oct 27 '11 at 13:07
3  
@slhck That's true, but people do flag answers when there's little else moderators can do besides those things. –  Bill the Lizard Oct 27 '11 at 13:17
add comment

I don't think too much weight should be put on flag weight if you decide to go with automatic actions. On SO, anyone with 15+ rep can get 500 flag weight in just 3 days and 600 in another day. And even reaching 750 would only take 2-3 weeks.

Also, if anyone flags it as an invalid flag I'd suggest that it should not be eligible for automatic handling and it should instead be dealt with by a moderator.

share|improve this answer
    
if someone spent a month getting to flag weight 750, by the time they were done, they would be a really good flagger even if they weren't when they started. And at that point, trusting their judgement is probably the right thing to do for certain kinds of decisions. The fact that, if you apply yourself, you can learn this skill quickly does not mean that it is a non-skill or that people who haven't put in a month or more of work learning it can do it as well as those who have. –  Kate Gregory Dec 9 '11 at 13:22
    
@Kate: I didn't think so much about if they'll know how to flag well as much as about the possibility for misuse. Spending a month or so to get 750 would probably be too much effort but if any kind of powers were to be given to users with 600 or less I think that it could be dangerous unless combined with other requirements (Rep, number of days on SO or similar). Otherwise it would be too easy for someone to create 5-6 accounts and then spend 4 days to get them all up to 600 which could then mean that they could delete answers as they want. –  ho1 Dec 9 '11 at 21:37
add comment
  1. The moderator queue tends to fill up with a lot of things that would have cured themselves if left alone for a few minutes.
  2. Questions that are marked as low quality which are often edited into better quality by the time a mod looks at the queue, for example. Ultimately the goal is to use moderators as a last resort where the community fails to do things, because otherwise they end up doing too much work (and might burn out).

Its not fair enough that the mod queue has become a way to push the burden of managing huge quantity of questions a day with a small bunch of working personalities.

share|improve this answer
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .