29

As you can see from Jay's answer we have some internal disagreements about how much signal we lose from displaying comments and, more critically, how much hiding that signal would cost. I waved my hands by suggesting that we will "increase our information density" if we hide more comments, but that's not a thing that can be easily measured or tested. So I'd like to ask what we might do to measure the cost of keeping comments around.

A good theory has three attributes:

  1. Testable—Specifically, it should be possible to set up a test that might demonstrate that the theory is wrong. This lies at the heart of the scientific method. Ultimately, split testing may be required. (An answer wouldn't need to actually do any testing, but it should show how a test might be setup.)

  2. Quantifiable—Generally, attaching numbers to properties makes testing something easier. If you want to test the ideal gas law you need to invent a thermometer first.

  3. Actionable—At the end of the day, a theory that can't be turned into a feature on the site is not really going to be helpful.

I have an idea (presented below), but what can you come up with?

  • 1
    Do you own an eye tracker? (Or know someone who could lend one?) – Flexo Dec 5 '13 at 18:53
  • 4
    I'm glad to see comments are getting so much attention these days after years of being considered "third class citizens". Is this your doing? :) – Shadow The Princess Wizard Dec 5 '13 at 19:53
  • 1
    @ShaWizDowArd: Yes. It's been my first big project on the job so far and something I've been thinking a lot about since August (of 2012 ;-). I wouldn't be pushing this so hard if I didn't think we could come up with a solution that's much better than the current system. As a company, we are getting more scientific in our approach to new features, which is very good. The cost is that we are probably going to be more methodical about changing things like how we display comments. – Jon Ericson Dec 5 '13 at 21:37
  • @Flexo: I don't, but I'm sure we could try using one if we had a good enough reason. How would you propose using eye tracking to test this? (I honestly can't picture how it would be useful.) – Jon Ericson Dec 5 '13 at 21:50
  • @Jon missed that old request of yours! Well, good luck with your project, hope we can come with something here soon. – Shadow The Princess Wizard Dec 5 '13 at 23:09
  • 1
    I was thinking of using an eye tracker to quantify time spend looking at comments vs scrolling time. If people don't dwell on comments much, but have to scroll past them to read content a lot then they just block valuable space. – Flexo Dec 6 '13 at 6:54
  • 1
    @Flexo Is this really a problem outside of meta (where arguably comments should be looked at completely differently)? Max of 5 comments are shown by default on an answer. Hopefully a given answer is longer than that, in most cases. I don't think scrolling past a bunch of comments is a problem (at least not on all sites. If you think it is on SO then I will acknowledge that you have better cause to know that than I do). – WendiKidd Dec 13 '13 at 3:35
  • The question uses the word "signal" twice, perhaps in mutually opposite meanings, for reasons of grammar or some semantic shortcut that I did not make out. Would you equate "signal" with "comments", or with "comment hiding"? That is, what quantity(ies) do you want to measure? – Jirka Hanika Dec 16 '13 at 12:02
  • Are you guys still looking at ways to clean up/hide trivial comments? Or perhaps ways to more easily clean out obsolete comments (mods don't like to touch them if they are over a specific length, even if the post has already been updated in response). Haven't seen any more posts in a while. – Troyen Dec 16 '15 at 23:39
10

If any sort of comment-hiding is enabled, I really hope that it will be something that can be opted into on a per-site basis and that the decision of whether or not to hide comments will be made on each site's meta.

A lot of what we're talking about here seems to be specific to Stack Overflow. So maybe SO needs this, maybe it doesn't; I'm not going to take a position on that. But on the smaller sites (and mine, ELL, specifically) comments are crucial. They're used to discuss improvements to answers (which are usually then edited in) and often there's a need to ask for clarification, as answerers tweak their answering to the needs of learners. Yeah, occasionally there are a couple chatty comments. And y'know what? I don't even delete those unless they get out of hand. On occasion, in moderation, they're fun. They're usually related to the topic in some way (we like interesting puns and English jokes over on ELL). A good bit of the fun of it for me would go away if there weren't the occasional not-completely-necessary comment.

There's not a lot of moderator time involved in cleaning them up; this is the kind of thing that isn't a problem when it's scaled down to a small site. When OT comments get out of hand we delete. That doesn't happen more often than mods have time to take care of it, for us. Also, comments are really how we get to know each other as a community. There's a lot of intelligent vetting of answers in comments, which is probably what makes me trust a given user's expertise equally as much as their answers.

So this was kind of a lot of rambling that I pieced together in several sittings ;) Might come back later and clarify things or add more. But I really, really would not want to hide comments on ELL.

  • 3
    +1. Comments really do help build community - at least on ELL - and allow native speakers to help non-native speakers with minor errors in how they are writing (e.g. in an answer to another question) without resorting to writing other answers or entering chat. To that end, comments on ELL play an integral rather than merely a supporting role on sites like ELL, and it would really hurt our site if they were made less easy to use or view. – Matt Dec 13 '13 at 14:19
7

I've found a flaw in my analysis.

It turns out that I ignored most posts with zero comments. The results including those posts significantly change the picture. I'm not sure that my theory is wrong, but I need to do some more analysis in the database. There's a really good chance I was thinking too fast.

Showing comments costs a post votes.

I believe that comments are most distracting to people who are reading a Q&A page. Comments are most useful for people who are authors or editors of individual posts. It's pretty easy to measure the output of authors and editors by looking at how often they create or revise posts. It seems harder to measure the output of readers on most sites. But on sites with voting (such as ours) the output is easy to measure: readers contribute by voting (up or down) on posts.

My guess is that when readers get to the bottom of a post, they tend to forget to go back to the top of the post and vote if there are comments to read. As a compulsive reader, I can say, anecdotally, that I'm far more likely to keep reading if there's more text. It's possible that comment clutter prevents people from going back to vote on posts without extra mental effort. If so, displaying comments might cost the authors of posts reputation.

That's a good story, but how can we test it? A split test where group A sees comments as they are now and for group B all comments are hidden would probably do the trick. If people in the test group vote (up or down) more often that people in the control group, we can be pretty certain that displaying comments is a drag on our reputation-based economy.

  • 13
    If your theory is correct, then long answers should also cost votes because by the time people get to the end they're far away from the voting buttons. Does the data say anything about that? – Monica Cellio Dec 5 '13 at 18:35
  • 4
    Comment: "This answer is no longer valid due to x...". This would naturally affect vote count, but not simply due to the distance to the vote buttons. – Geobits Dec 5 '13 at 18:38
  • 1
    sort of test could be like, pick two groups of questions from the history (data dump): one having more comments, another having less, with question score and answers size close enough. Find if there's a correlation with amount of votes – gnat Dec 5 '13 at 18:40
  • Longer posts also trigger the tl;dr. That might be hard to sort from the "I forgot to upvote". – Mysticial Dec 5 '13 at 18:45
  • 2
    Does it matter if comments cost people reputation? I'm not sure I care, though lower reputation users might more. – ben is uǝq backwards Dec 5 '13 at 18:48
  • Well, some good points here. Maybe we want the ^v buttons to be floating with scrolling, as long as they fit next to the answer. (Hint: No, we do not. However, maybe the fact that once you finish reading a long answer you can't see the upvote button is bad, and should be changed somehow.) – yo' Dec 5 '13 at 18:51
  • 4
    What proof is there that the relationship is not in the other direction? More comments means more people attempting to help the OP because the question isn't comprehensible. Can you split this by reputation at the time of asking? (unless you have a better indicator of question asking ability - in which case that'd be better 'cause this ones not great) – ben is uǝq backwards Dec 5 '13 at 18:52
  • @benisuǝqbackwards: Note that the query is over both questions and answers. – Jon Ericson Dec 5 '13 at 19:01
  • 5
    In my opinion, more comments indicate less consensus. The lower amount of consensus the less likely to get upvotes. I don't think that comments negatively affect voting by just being there, I think that comments which point out flaws with the post contribute to the way that post is viewed, especially if their claims are either undisputed or undeniably accurate. – Travis J Dec 5 '13 at 19:12
  • 4
    I'm not sure that I agree with your analysis of the relationship between comments and votes. A question that needs clarification often gets a comment. All custom close messages are comments. Trying to point out where something is wrong in an answer (often in the critical early visibility) is a comment. All of these are issues that cost votes and comments are the symptoms of the problem (low vote scores are also a symptom), not the causes. – user213963 Dec 6 '13 at 3:11
  • 4
    "Any algorithm for displaying comments should err on the side of hiding comments." - Wait, no! Sometimes a comment points out a critical flaw in an answer (e.g., it points out that the answer is wrong or bad advice). Erring on the side of hiding comments would be harmful. Giving answers more upvotes is not our only goal. We want to give good answers more upvotes, but give bad answers fewer upvotes. I don't think we can make a blanket statement that hiding comments is always the safe choice; it's not that simple. – D.W. Dec 7 '13 at 4:29
  • Also, questions seem like they might be pretty different from answers. Merging them together into a single pool seems like it could lead to misleading conclusions. – D.W. Dec 7 '13 at 4:30
  • After reading all those comments, I wanted to move to the next answer straight away to see what it might be. Almost forgot to upvote that one. – Szymon Dec 8 '13 at 2:44
  • 1
    Why don't you make it a user (i.e., reader) preference? Maybe even let users set a comment-hiding threshold (default value). And then give them a button on each page, to hide/show all etc. --- buttons to expand/collapse a list are commonplace. IOW, why do you need to come up with a single best-fit design/solution that you hard-code? Different people browse & read differently. – Drew Dec 13 '13 at 2:11
7

We are running an obsolete and chatty comment debt.

Anecdotally,1 very few of the extant comments on Stack Overflow (or any other Stack Exchange site) are rude, offensive, or spam. That's a testament to the rousing success of our flagging system. Over the years, the rate of rude or offensive comment flags to new comments on Stack Overflow has stabilized2 to between 0.1 and 0.15%:

Helpful rude and offensive flag rate by month

The absolute number of rude and offensive comments posted to the site has increased as time has gone on, but flagging of such comments seems to have kept pace easily enough. There is no particular problem with excessively rude comments on our sites. However, when I read questions on the site, there's no question that many of the comments I see are obsolete or chatty. My subjective estimate is that somewhere between a quarter and a third of all comments have been overtaken by events or are little more than social pleasantries. Here's how the flag rate on those comments has progressed:

Helpful obsolete and chatty flag rate by month

Note that the heavy spike of activity starting in September of 2011 coincides with the introduction of several flagging-related badges on August 28 of that year. The spike is a testament to the temporary power of extrinsic motivation. It also shows that there were plenty of existing obsolete and chatty comments available for people to flag. (Note, that all data includes helpful flags only.) The drop off in flagging rate likely has more to do with people earning their badges and no longer feeling the need to flag such comments.3

In the last year or so, obsolete and chatty flagged comments have increased from about 0.1% to nearly 0.5% last month. Even so, that seems subjectively at least an order and a half of magnitude off from the actual rate such comments are being added to posts. Assuming commenting has not improved substantially, we are racking up a considerable debt of obsolete and chatty comments. Here is the number of new comments and comment flags being added each month4:

Comments and flags by month

Why should we care?

The trouble with this theory is that it doesn't seem like a big deal if you happen to think these comments aren't such a big deal. But there are two reasons we should be concerned:

  1. Broken windows

    Many people see these sorts of comments as unsightly and encouraging of more of the same. The current system leaves most people no option other than flagging to clean up trivial comments. That results in a sense of powerlessness which is very unusual on a Stack Exchange site. Even if you disagree that there are broken windows, you have to agree that some people are annoyed by trivial comment clutter and (because of the deficit we are running) will never be satisfied. We have a system (comment flags) that barely scratches the surface of the problem. Flagging rude comments works. Flagging trivial comments does not.

  2. Moderator time

    Last month, we saw ½ million posts with a million comments. There were 188,109 total helpful flags of which 9,985 were on comments:

    3722 Comment Obsolete
    3618 Comment Not Constructive Or Off Topic
    1179 Comment Too Chatty
     990 Comment Rude Or Offensive
     476 Comment Other
    

    So if you don't think keeping obsolete and chatty comments visible is a big deal, moderators are wasting substantially more time dealing with trivial flags than with flags on outright rude comments. No matter how you feel about trivial comments, we are asking our moderators to clean the Augean Stables with teaspoons. Ought we not free them up to focus on important flags?

Testing a comment hiding scheme.

When testing a comment hiding scheme, count it success when the test group raises:

  1. Fewer obsolete and chatty flags, and
  2. No change in the rate of rude and offensive flags.

Not constructive or off topic and Other flags are ambiguous. It may be that simply hiding rather than deleting such comments will be an acceptable result. I would suggest that comments would need to be displayed for at least a day or two in order to satisfy the second condition.


1. Which is to say, in my subjective opinion.

2. There are two spikes in the first half of the graph. The first, I believe is the natural enthusiasm that comes from a new button to push. The second is likely related to the improved flagging introduced during January of 2011. The substantial drop off in flagging during May, 2011 is likely a result of splitting comment flags into several new buckets, but I don't immediately see where (or if) that was announced. Obsolete and chatty flags begin in that month.

3. I believe that I can verify this by comparing the number of people who earned the badges to this graph. But I don't think this is a critical part of the argument. Even if badges had nothing to do with this spike, the fact that these flags have not stabilized is a red flag (so to speak).

4. I'm including all helpful comment flags on this graph since comment flags seem not to have been categorized early on. Flags and deleted comments are not exported to the public data dumps. But you can verify the basic shape of the comment graph using this query.

  • 2
    Rather than changing the visibility rules, couldn't you alter the deletion process? As far as I can tell, comments will be deleted either by the poster, a mod, or after {enough} flags are raised. {Enough} varies by flag type, and the exact numbers seem to be a secret, but can't it be tweaked? If several users agree that a comment isn't worth keeping, it probably isn't. It seems to work for rude/offensive, so why not apply the same to obsolete/chatty? – Geobits Dec 13 '13 at 2:17
  • 1
    @GenericHolidayName Of course, there's also a lot of obsolete/chatty comments lying around, even on the old questions that get lots of google traffic. That's dumping a huge amount of manual work onto users - something that would be far bigger than the SO close vote queue. – Troyen Dec 13 '13 at 8:02
6

I have found comments to be tremendously useful, but at the very least you could say that comments are a sacrificial metal, acting to protect the purity of questions and answers by giving people a place to post... well... comments. Hiding them would significantly reduce their effectiveness in deterring poor edits and low quality answers by the people who would be least likely to find them... new users. How can you possibly calculate the "cost" (or value) for that?

5

The risk of hiding comments?

Rarely are there comments which on their own significantly add to a post. The entirety of the comments may add content, but usually only one will shine through as the comment which caused the post to become more significant or contextual.

Hiding comments is an easy one click fix for a user, so the real risk is that a high value comment hidden would have been overlooked because the user did not know it was there.

As a user looking for value in posts, comments are often a place to look when examining the critique of a post. If there are no critiques or suggestions of value, then there is no point in examining them so I think that as long as enough are shown to entice a user to click through for the entire set then that will be enough to allow users to find the high value comments if they are hidden.


Measuring the risk

When being thorough, a user may want to see every comment regardless of their quality. I don't think many posts actually require the entire set of comments to have the answer be relevant.

Split test

Test for the amount of interaction done with the comments where finding them by clicking is undesirable, and upvoting them is desirable.

I think a measurement could be in the form of click through comparisons plus upvotes. Group A sees the status quo, B sees the new version (whichever version that is). Compare total points from how many times each group triggers the reputation hover for the username of a comment (+1 per username), upvotes a comment(+10), expands all comments(-5).

Whichever group has the most points indicates the most positive interaction with the comments.

  • 1
    Can you explain a bit more about how you came up with the values for the various actions and why you think they are useful indicators? I'm a little confused about the "+1 per username". What do you mean by that? We've talked extensively about what signal a user is sending when they upvote and expand comments. (Expanding comments is especially confusing when the action is also tied to writing a new comment as is the case right now.) But I think you are on the right track. – Jon Ericson Dec 7 '13 at 1:29
  • 2
    Your proposal does not measure anything about the extent to which the comment helped improve our evaluation of the answer or of the question, nor how often it triggered some improvement in the question/answer. (For instance, comment causes someone to edit the question/answer: you give 0 points. Comment causes someone to upvote good answer they otherwise wouldn't have: 0 points. Causes you to downvote bad answer you otherwise wouldn't: 0 points. That's odd.) And why is expanding all comments a negative interaction? The point values feel arbitrary and not clearly connected to any end goal. – D.W. Dec 7 '13 at 4:37
  • @Jon - It was my understanding we were determining a metric to measure the impact of hiding comments. If hidden, the comments can be found with 1 click so it is not like they are removed - and as such I do not think hiding their content will change their affect on the post. I believe the only affect this will have on users is time. Extra time from inspection. The weighting came from that: Did the user have to click through or heavily inspect comments? Because if they do that often, there is no point in hiding them. – Travis J Dec 7 '13 at 20:17
  • This is why if an expansion click occurs, it should negatively affect the interaction (since users probably didn't need that content hidden), if they upvote a comment it should positively affect the interaction since they found a useful point, and if they hover a username (which shows reputation) it should show that they are interested in the source of the content. The weighting is biased heavily towards upvotes, acknowledges hovering inspection, and biased against expanding. – Travis J Dec 7 '13 at 20:18
  • @D.W. - The points are based on my assumptions because finding the scientific exact numbers to provide a perfect statistical model would be overengineering. This is in response to the cost to the user of displaying comments. The cost to the post? There is no direct blanket relation in my opinion so I did not cover nor address that. – Travis J Dec 7 '13 at 20:20
4

Let's start from the basics.

  1. Comments suck up moderation resources and reader time for potentially dubious value
  2. Comments are really really hard to evaluate because of how they're used/how they need to be looked at
  3. Despite being a complete hassle to deal with, comments play a very important role in clarifying/improving content that there is no other tool for

Because of these three issues, it is exceedingly difficult to get a clear signal from what we have now, which is why we are in this situation in the first place. Rather than trying to measure the signal we have, we should add more information to help clarify that signal, or potentially look in to changing the comment system to better reflect the different roles that comments actually play on posts.

The Problem

As Jon points out there were almost 100,000 comment flags handled by mods in a month. That's a lot of flags. As someone who's deleted several hundred comments in my corner of SE over the past month, I can tell you that comment flags are my least liked of all flags because they:

  1. Need to be read in context (you have to read the entire question, answer, and the entire comment thread to determine what the heck to do as a result of the flag)
  2. Have varied actions that need to be taken (you need to delete, or edit the comment, or edit the comment in to the post, or purge the whole lot of comments because it's two people bickering, etc.)
  3. Sometimes are just great signposts that don't require any action but have no alternative way of being handled (for instance linking to a related non-duplicate question for a newer version to fix the same issue, etc.)

This has all been stated before, but from where I stand this means we have to focus on two separate (but related) problems:

  1. Measuring if a comment has value
  2. Measuring what the value of that comment is

Getting a clear signal

Right now we have three different things we can measure in regards to comments:

  1. Comment upvotes
  2. Comment length/content
  3. Comment deletion (by author or mod or flags)
  4. Comment responses (through @mentions)

As Jon stated:

People aren't perfect: I've observed that votes on short comments tend to mean "Funny". But votes on longer comments tend to mean (to take a page from Slashdot) "Insightful", "Interesting", or "Informative". Comments that we want to keep around have a combination of length and upvotes.

So the best signal we have requires a combination of factors. An upvote doesn't mean useful content, it means 'potentially useful content' that has to be correlated with other data which gives a signal that isn't clear enough to actually act on. What we need is a simple indicator of whether the comment has value in regards to what comments are supposed to be for:

  • Request clarification from the author;
  • Leave constructive criticism that guides the author in improving the post;
  • Add relevant but minor or transient information to a post (e.g. a link to a related question, or an alert to the author that the question has been updated).

That signal does not currently exist, so we need to make one.

The easy way or the hard way

I see two ways to handle this. One is to create a weaker signal (but require less effort from users meaning more participation and a broader sample), and the other is to create a far stronger signal (but require far more effort from users reducing participation and potentially introducing sample bias).

The Easy Way

You create a one-click button next to every comment:

Comment Value Clicker

Participating could be limited to people over X rep, and optionally require participants to read a page summarizing what we want them to click the button for (comments that do not follow the comment guidelines). These clicks get recorded in the database somewhere, and allow us to get a clear signal of which posts users think provide the right signal according to our guidelines, and which don't.

We can view what actions users take after clicking it (do they flag? add a comment? edit the post? vote?) for further analysis, and we can look at the types of users clicking the button (heavy commenters? heavy flaggers? heavy editors? browsers? answerers? askers?) to get a better idea of what those clicks actually mean (versus what we hope it means by starting the entire process in the first place).

By using these guinea pigs volunteers, we can then test a comment-hiding algorithm on them to see how different types of actors react to the changes (do they click the 'expand comments' more? less? what types of users are using comments in which way?). This prevents any loss of signal to the community at large while still getting us some good data we can use to figure out what the heck we should do.

Regardless, this would be really analysis-intensive on the SE side since the signal would be ambiguous at best (it shares almost all the same flaws as the comment upvote feature).

The Hard Way

On the other hand, we can try to get more detail feedback categorizing posts. Using a similar volunteer method, we can give folks the ability to curate comments on their own posts by indicating what type of content is in the comment. Either creating some sort of a review queue for comments on your own posts, or having buttons next to the comments like in the other suggestion, we would want our guinea pigs volunteers to tell us if a comment is:

Clarification Constructive Criticism Minor or Transient Information Other (but good to have) Noise (Delete)

By limiting this to comments on your own posts you are likely able to get more participation (because I'm going to wager that people click through to their posts whenever a comment is added), and the information that you get will be a lot more detailed (on a more limited set of comments). This will make the analysis on the SE side far far easier, but due to the added burden for the folks actually participating, you may end up skewing the sample willing to leap through the hoops, and hurting the value of the information to more casual users.

This also has the benefit of seeing how people would be at moderating comments on their own posts, and act as a good test group to see if that sort of privilege is something that would have benefit, or would be ripe for abuse (with users deleting any comment that doesn't compliment them or other not-so-great behavior).

Either by doing the hard or the easy way, these groups could be used to test community-driven comment moderation and gather more data on how the entire process could be resolved, not only limited to hiding comments, but also to deleting them without significant moderator intervention. Either by letting folks moderate comments on their own posts, or by giving them the ability to mark bad comments and have them auto-hid/auto-deleted in the future.

1

Cost of displaying comments is the risk of obscuring valuable information. When an important comment is buried under 3-4 chatty ones, there is a high chance for the reader to miss it.

Another cost of displaying comments is that this misleads readers into thinking that these are first class citizens at Stack Exchange. Meta posts and help center articles can repeat comments are ephemeral until the hell freezes over, but tens, hundreds thousands readers seeing 3-years old chit-chat are presented with a fairly strong evidence for the opposite.

From this perspective, the more comments are hidden, the clearer is the signal sent to site visitors: "yes, we really mean it, comments are ephemeral".

Consider stressing the educational aspect of your proposal. For this to work though, there should be a consensus within SE dev team that 1) ideally, valuable information should be eventually moved from comments to posts (via editing or new posts) and 2) that additional measures are desirable to educate and incentivise site users about that.

It is important to note that focusing on educational aspect opens a door to compromise with those opposing the proposal. Think about it, if vast majority of views on "matured" questions fall at those with "compressed" comments, you can afford some relatively minor amount shown the other way.

A possible compromise could look like this: hiding comments is disabled for 3 (5, 7) days if the question gets 10 (100, 1000) views a day, or if it gets 50 (500, 5000) views a month (later is for posts with longer attention span which have lower count of views per day). Numbers here are only for example, what you end with may be different - you only need to ensure that sufficient majority (70-80-90%) of views at matured questions fall at either those without comments at all, or at those with compressed comments.

This way, you basically "trade" two risks against each other - a risk of auto-hiding valuable comments against the risk of misleading site visitors about the ephemeral role of the comments.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .