-28

The problem

There are a huge number of noisy comments, and they generate a huge number of flags. But, a large number of flags are raised on comments where they won't do very much good: if there's only a few comments on a post, removing one of them doesn't make an appreciable dent in the noise.

Here's a fun chart:

% of comment flags by number of comments on a post

The x axis is the total number of comments on a post (deleted or visible). The yellow line is the fraction of moderator-actioned obsolete and too chatty flags raised on posts with this number of comments. The blue line is the same, but cumulative, i.e., the fraction of such flags raised on posts with this number of comments or less. Data is for Stack Overflow during the past 356 days.

As you can see, over 50% of "noisy" comment flags are raised on posts with 5 comments or less. On average, that's well over 100 flags every day. The raw numbers are smaller on most other sites, but the percentages are pretty much the same (with the exception of a handful of sites where pretty much any comment turns into a long discussion/argument - you know who you are).

Now, don't get too fixated on the numbers here; the real cost is probably a lot higher, since some folks use "not constructive" for literally everything and I'm not counting those. My point is simply that a majority of these flags are chewing up moderator time in situations where they're not actually doing any good.

The solution

First, we need better data on this. Re-working the wording of comment flags (and dropping redundant options) should help there.

But then... I suspect we can easily get away with just ignoring "noisy" comment flags (those that don't indicate rudeness or some other serious problem) on posts that have fewer than 5 comments in total (deleted or visible).

To that end, we should hide these flags from the moderator flag queue when there are no "rude" or "other" flags on the same post, and when the post has never had more than 4 comments on it. Focus moderators' attention on getting rid of serious problems and removing noise when it actually obscures signal. The flags can hang around until more comments are posted or until other flags are raised... Or they can hang around forever, doing no more harm than the comments themselves.

See also: Drop "not constructive", combine "noisy", reword "rude" and "other" comment flags

  • 7
    What about letting x k users help handle some comment flags? I forget where the meta post about that is, but it seemed like a good idea to me. – ɥʇǝS Mar 8 '17 at 0:11
  • Building anything to handle that is punting this down the road another 2+ years, @ɥʇǝS – Shog9 Mar 8 '17 at 0:13
  • Fair enough, but is it still "in the works" or being considered? (these changes seem fine regardless, but I'm curious) – ɥʇǝS Mar 8 '17 at 0:18
  • No. Comment flags are a complete mess right now; the amount of work needed to sort them out into a review queue, train folks to use it, construct sane audits, build a system of oversight (remember, non-mods can't even see deleted comments)... Is staggering. – Shog9 Mar 8 '17 at 0:20
  • I don't quite understand what that graph is showing, but wouldn't it be more helpful if it included data for helpful/declined ratios on such comment flags? I mean, if flags on short comment chains are consistently being declined, then you might have a point, but I suspect that isn't the case. – Rand al'Thor Mar 8 '17 at 0:21
  • Here's the thing, @rand: I don't want to have to decline these flags; most moderators don't want to decline them. We need people flagging noise, rudeness, etc. because we've no other way to get rid of it; the last thing we want to do is discourage anyone. But... That doesn't mean it's an immediate problem either. – Shog9 Mar 8 '17 at 0:28
  • 16
    The logic here seems backwards: If there's only a few comments on the post, removing one of them makes a huge dent in the noise, as in, can eliminate 100% of it with a single comment. – Jason C Mar 8 '17 at 1:33
  • 1
    Also this seems strange, but maybe I'm misunderstanding. Essentially this seems like putting a "pressure release" valve on comment flags on new (hasn't had 5 comments yet) posts, no? So rather than taking care of potentially volatile situations right when they start, instead you'd wait for it to get 5 comments deep in a non-constructive conversation? I mean I guess the "rude" criteria helps here, but... it seems weird... – Jason C Mar 8 '17 at 1:34
  • 11
    As a user, not a mod, it feels wrong that a flag I raise could fall into the bit bucket for any reason. I'd rather have a flag declined as not useful than pending for several weeks. I operate on the premise that as a member of the community part of my responsibility is to report, to the mods somehow, when I see things that aren't proper. As a new member I try to learn what's the proper way to handle issues by reviewing how previous input was received, and/or acted on. The idea of passing some flags to the community for review, however, seems valid. The First Post and Late Answer queues work? – Gypsy Spellweaver Mar 8 '17 at 1:35
  • 9
    @GypsySpellweaver Easy workaround: Just flag chatty/NC comments as rude or custom instead, then sit back and watch the mods facepalm at all the new incorrect flag reasons. Even if users don't know a mechanic like this is in place I suspect a fair amount will learn over time that their rude/custom flags are getting handled more quickly, and give up on the others. – Jason C Mar 8 '17 at 1:39
  • 2
    @JasonC Mods have a hard enough job, last thing I'd want to do is add to the load needlessly. If, OTOH I can make it easier, either by choosing the correct flag, or by joining in the queue pre-review process, that I'm for. I've already earned the enmity of a mod on one site for raising his presence on another site. (Not that he's said it, but I know he should.) – Gypsy Spellweaver Mar 8 '17 at 1:45
  • 1
    @GypsySpellweaver Just don't be overly picky about comments. A single "thank you" comment under one answer? Really no reason to flag that. – ɥʇǝS Mar 8 '17 at 1:47
  • @GypsySpellweaver Oh absolutely. I was being cynical about possible unintended consequences of a change along these lines. – Jason C Mar 8 '17 at 1:48
  • @ɥʇǝS That wouldn't get a flag from me anyway. Per guidelines they don't belong, I know. Still, it seems like an extra, personal, gratitude from querent to respondent beyond the rep of an accept. If it does get deleted later, fine, the sentiment has still been delivered. Those are ones I'd imagine a mod might clean up while attending to other issues in the question anyway, and not worthy of direct action. – Gypsy Spellweaver Mar 8 '17 at 1:53
  • 6
    Let a machine handle these second grade citizen's, no need to spend human effort. If flagged run it though the algorithm, if hit just deleted it, I bet not much useful information will be lost and we can keep it nice and clean – Petter Friberg Mar 8 '17 at 8:31
36

Comments aren't any more useful just because they're few in number.

Here's a few examples of short comment threads which I would flag on sight as a non-mod and delete as a mod, and on which flags are very easy to handle:

  • (on a post by UserA)

    UserB: You've made a typo in the second paragraph: [...] should be [...]
    UserA: @UserB Thanks, fixed!

  • (on an answer to a question by UserA)

    UserA: Thanks, nice answer! +1

  • (on a highly-upvoted post, or any post really)

    UserA: Why the downvote?

Handling "obsolete" or "not constructive" or "too chatty" flags on these takes a matter of seconds, so they're not even expensive in mod time. Why should those comments have to stick around just because they're the only comments on a post?

People might use the 'less serious' flags even for serious issues.

If a comment is "rude or abusive", it's also likely to be "not constructive". Faced with a choice of different flags to use, people won't necessarily go for the "rude or abusive" one - maybe because it feels too harsh, maybe because they like to use the "not constructive" flag whenever possible, maybe because they tossed a coin and it came up tails. I don't want to see offensive comments left in place just because people used the "not constructive" flag option and no moderator ever saw them.

Flags shouldn't be left pending forever.

Partly this is a philosophical objection: the whole point of flagging is to get moderators to look at things which need their attention. If certain flags never actually have a chance of being shown to mods and getting handled, why let people raise them at all?

On a more practical note, when people check their flag history and find old comment flags still pending, they might either take it as a sign that mods are being lazy or inactive, or go and complain on meta that their obviously-valid flags haven't been handled. Such meta complaints might actually end up taking more moderator time than just handling the damn comment flags would have done.

  • 1
    This is why doing anything here is predicated on getting rid of ambiguous flags - as long as folks are using the same flag for gratitude and name-calling, it's nearly impossible to do anything clever. And speaking of gratitude... Most questions get at most one answer. Most answers get at most one comment. If that one comment is "thanks, nice answer"... is it really doing any harm? – Shog9 Mar 8 '17 at 0:26
  • 1
    @Shog9 Even with your proposed revamp in that other meta post, I could still easily see people flagging offensive comments as "no longer needed". Because a comment that says "f--k you" is no longer needed, even if it's also rude or abusive. – Rand al'Thor Mar 8 '17 at 0:29
  • 1
    Then we'll change the wording again, @rand. And again, and again, until we find something that mostly works. – Shog9 Mar 8 '17 at 0:30
  • 1
    This is the fourth comment, so it would stay here indefinitely if flagged under the proposed system. – Wildcard Jul 25 '17 at 2:25
  • @Wildcard: indefinitely, or until some clown comes along and adds a fifth comment.     :-)     ⁠ – Scott Jul 30 '17 at 1:44
23

I don't like the idea of ignoring flags. It sends the wrong message to users that are trying to help clean stuff up.

I propose a different solution: Automate comment removal. I've been automatically flagging comments for removal since July 2014. Of the 238,966 comments flagged on Stack Overflow in 2016, 35,610 are mine. Nearly 15% of all comment flags, which is actually down from 2015. I am a big part of Stack Overflow's comment flag volume. But, I've been accurate. In 2016, I had 12 declined comment flags.

My system isn't as complicated as other recently shown automatic tools. I have a simple Naive Bayes classifier. I only flag things that the classifier is very sure is "too chatty" or "obsolete". I don't do anything with "not constructive" for may of the same reasons you've outlined.

Last night I adjusted what I flag to only flag comments that have multiple siblings. Previously, I wasn't considering the number of comments on the post, but based on this proposal and further discussion, it makes sense to focus on the noisier posts. It's not like Stack Overflow is hurting for comments either.

Filtering the comments doesn't need to be at the time of creation. As you mentioned above, a "thank you" comment isn't hurting any one. But, come through a day or two later and clean up these old comments automatically. Right now, I'm looking at comments that are 48 hours old or older. There are plenty, every day, that can be cleaned.

What would it take for Stack Exchange to implement this type of comment clean up on your side?

  • 1
    BTW, your work may skew Shog data somehow. I think his graph would be better if it excludes outliers flaggers. – Braiam Mar 8 '17 at 16:26
  • I don't think excluding outliers is the right thing to do. Outlier or not, I contribute to Stack Overflow's comment flags as much as the person who only flags one comment. All of that has to go past a moderator for handling. Ignoring those flags doesn't solve the issue. In effect, it just that would just make those flags elevator close door buttons. A better way to handle those flags is the end goal and I think an automated system will help with that goal. – Andy Mar 8 '17 at 16:36
  • 7
    At the scale you're operating on currently, I could run this locally & keep up with comments on Stack Overflow, which might be the better option here - building this in would require training it on each site where it was needed, which would probably take forever on all but a half-dozen sites. Something I've been mulling over since you announced this; quite honestly, I just kept hoping you'd be elected moderator & run it yourself ;-P – Shog9 Mar 8 '17 at 17:18
  • 4
    @Shog9 PULL HIM UP! What are you waiting for? :D – Bhargav Rao Mar 9 '17 at 10:28
  • 2
    Feels kinda self-serving, @Bhargav. Also, you're handling 100-200 comment flags a day currently, which gives us a bit of breathing room. – Shog9 Mar 9 '17 at 15:21
21

Why not remove spam flags until there are 5 or more, while you're at it?

Comment flags do indicate a problem. Something must be done. Hiding comment flags means that the problem remains on the site and every visitor suffers from it. That's no good.

We might then work around that deficiency by creating a chatroom where people post links to comment to flags, in order to reach the threshold where action will be taken. It used to be necessary to do this to get the requisite 5 votes to get a post closes; then the close review queues automated this on most sites other than SO. And now we'd have to do this for comments?

Or people might get into the habit of flagging obsolete comments as “needs moderator attention”, just so that moderators do notice. Which I'm sure will lead moderators to decline the flags, leading to more whining on meta. And the whiners would have a point: if an “obsolete” flag is basically ignored, then the only way for a non-moderator to request the deletion of a comment is to do whatever it takes to summon a moderator!

This change only means uglier sites and more work for people other than moderators. As much as I feel I'm wasting my time handling obsolete comment flags as a moderator, I don't want the alternative of making everybody else waste their time on useless flags or convoluted flagging processes.

  • Same question I asked rand: Most questions get at most one answer. Most answers get at most one comment. If that one comment is "thanks, nice answer"... is it really doing any harm? – Shog9 Mar 8 '17 at 0:47
  • 3
    @Shog9 Not particularly, no. But the chances that it gets flagged is also low, so that's rather a strawman argument. – Gilles Mar 8 '17 at 0:48
  • 9% of all obsolete/chatty comment flags on SO are on answers with a single comment. By the time you hit 5 comments - the point where potentially removing a comment could do some good by making room for a new one - you're at 42%. CS gets as many of these flags in a year as SO does in three days, so not the top of my list when I'm analyzing this stuff. – Shog9 Mar 8 '17 at 0:58
  • 7
    @Shog9 No, it's not doing any harm - but equally, if it does get flagged, does it do any harm to delete it? Ignoring flags strikes me as a good way to get (rather justified) meta complaints. If you were talking about removing the possibility for flagging such comments, I might be able to get behind that; but letting valid flags go into the void sounds like a bad idea all round. – Rand al'Thor Mar 8 '17 at 1:03
  • 2
    @Shog9 9% won't save you much. That feels like useless optimization. That being said, if the volume of comment flags is such a burden on SO, maybe the way the system works needs to change on SO — but do consider the consequence of changing the rules, instead of asserting that the change would be harmless. – Gilles Mar 8 '17 at 1:04
  • @randal'thor The harm they do is additional workload for moderators. Since that's a hard problem to solve (setting up a review queue is not trivial and means more work for other people), Shog9 proposes to ignore the harm that leaving the comments around does. – Gilles Mar 8 '17 at 1:05
  • 9% won't - but 40-50-60% would. If we can establish a threshold of necessity here, then we can prioritize flags accordingly. – Shog9 Mar 8 '17 at 1:15
8

I don't think this is a massive time sink for mods.

Anecdotally, a good chunk of these flags are valid - either small edits that are needed, actual noise and take up minimal moderator time to decide on a correct course of action.

I like the related idea but quick cleanup of un-necessary flags feels more efficient than leaving them there until it blooms into a thread of comments, many of which are un-necessary.

The streamlined flag system does seem like it would improve triagability but for most sites, this feels like a non issue. The 2-20 seconds I spend looking at a comment dosen't add up all that much, least in my experience.

If we must offload these - from the (and I use a dirty word here) cultural standpoint - why not make a queue for these? These flags are anonymous anyway and this feels like the sort of thing the community can handle. That said, at SU volumes of comment flags, except on a quiet day, the benefit of this seems minimal.

Alternately, have it opt in per site. SO makes a great guinea pig test bed, sure but letting mods pick whether they have this would likely keep most folk happy.

  • 1
    Shog already mostly addressed this in a comment reply to me here. – ɥʇǝS Mar 8 '17 at 1:02
6

What you're proposing is not a solution. You're simply hiding the existing problem (too much work for moderators) and creating an additional problem that we had in the past: flags stuck on pending for months/years. Do you want to go back to people asking why their flags are stuck for so long?

  • 1
    Quite honestly, I'd rather have the system automatically handle these whenever possible; that's how it was constructed originally. Even now, the system handles more comment flags than any single moderator; a huge number of flagged comments match some fairly simple patterns. Problem is, we throw these in front of the moderators ASAP, and some heuristics are pointless as long as that happens, chiefly age and total comment count (which also tends to be a function of age). Given the concerns raised here, probably won't ever do this network-wide though. – Shog9 Mar 8 '17 at 17:14
6

Make "noisy" comment flags lowest priority in the queue instead of hiding them

I agree that there is little harm in keeping a comment that says "thanks" or the like, especially if there are so few other comments on the post that none are being hidden. That said, I'm philosophically opposed to hiding any kind of flag.

Instead of hiding a "noisy" comment on a post with fewer than 5 comments, the system can heavily de-prioritize it in the moderator flag queue since it's not urgent. The moderator flag queue is already prioritized by type of flag (among other things), as explained on the flag queue page:

mod flag queue info

In addition to prioritizing the queue based on flag type (e.g. rude flag, not an answer flag, comment flag), it can be further prioritized based on characteristics of the flag within that type: a comment flag on a post with fewer than 5 comments can be de-prioritized compared to a comment flag on a post with 5 or more (since there aren't enough comments in the former case to cause any comments to be hidden).

In my experience as a moderator on Science Fiction and Fantasy these kinds of flags don't take up too much time and our mod flag queue is often empty anyway -- constraints on the moderators' time isn't an issue for us. These kinds of flags are a problem on larger sites like Stack Overflow, but they won't distract the SO moderators from more pressing issues if they're de-prioritized. If the SO mods never get to them because their queue is always filled with more important issues then it's no big deal and it's the same result as hiding them. But on the sites where the moderators can keep the queue empty these flags are still dealt with.

To further de-prioritize such comment flags, they could even be put into a secondary mod queue so that a moderator who doesn't have time to look at them doesn't even have to see them. Or they could be set to not increment the mod flag count in the top bar. (Personally, I don't think these are necessary, but they're other possibilities.)

Finally, if the moderators are really so pressed for time that they can't properly handle all the flags in the queue, why not elect more moderators?

  • The subtle problem here isn't that the flag type is too prominent (it isn't) - it's that there's no prioritization within the flag type: a lone answer with one flagged comment ranks the same as a question with a dozen comments, one of them flagged. Something we've already done is couple comment flags with the "too many comments" auto-flag, to highlight posts where a conversation is going south quickly - so my fall-back idea here is to build on that, escalating comment flags to post flags in cases where there are many comments. – Shog9 Mar 8 '17 at 17:22
  • 1
    @Shog9 I'm saying that there should be prioritization within the "noisy" comment flag type: if the flag is on a post with fewer than 5 comment then it is de-prioritized compared to flags on posts with more than 5 comments. – Null Mar 8 '17 at 17:25
  • Right, there's no support for arbitrary flag priority within a type now. – Shog9 Mar 8 '17 at 17:58
2

Why not let non-moderator users at a certain rep threshold (20k, for example) delete/vote-to-delete comments that contain the word(s) "thank you" or some variation ("thanks" "thansk" "tnx" "10x" "hanks" "thaks", etc), if the comment has been flagged as "Not needed (anymore)" (or a separate 'noisy' option)?

You can add the list of them to the 10k moderators tool page and have it visible only to 20k users.

These comment flags would only make it to a moderator queue if they were not cleared within 7 days after the last action (ex: a comment is flagged for this, someone votes to delete it 6 days later, and a second person casts the second, final delete vote on it 6 days after that; the flag would never be seen by a moderator).

Delete comments are still visible to moderators, so in the event of abuse or a meta post, mods could easily go to a post where comments had been deleted and see who has been deleting what comments.

And in keeping with current practice, comments deleted on the same UTC day that the delete vote was cast would mean that delete vote gets refunded to the casters.

1

I don't know how this plays out for SO, but on smaller sites it works a lot better if somebody flags a comment stream early at the first sign of it going off track. I'd much rather a flag on the first comment that starts to veer into debate rather than get paged in when there are 25 and three users are all fired up at each other.

By the same token, the resulting impact of a flag handled early is much greater. People mimic what they see, and if we allow comments that would otherwise be deleted to proliferate up to 5x per post, our site is going to be a marshland ... not as deep as a lake but even harder to navigate in a boat.

Comment flags should not be handled any differently based on how many comments are on a post.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .