How can we measure the cost of displaying comments?

Question

As you can see from Jay's answer we have some internal disagreements about how much signal we lose from displaying comments and, more critically, how much hiding that signal would cost. I waved my hands by suggesting that we will "increase our information density" if we hide more comments, but that's not a thing that can be easily measured or tested. So I'd like to ask what we might do to measure the cost of keeping comments around.

A good theory has three attributes:

1. Testable—Specifically, it should be possible to set up a test that might demonstrate that the theory is wrong. This lies at the heart of the scientific method. Ultimately, split testing may be required. (An answer wouldn't need to actually do any testing, but it should show how a test might be setup.)

2. Quantifiable—Generally, attaching numbers to properties makes testing something easier. If you want to test the ideal gas law you need to invent a thermometer first.

3. Actionable—At the end of the day, a theory that can't be turned into a feature on the site is not really going to be helpful.

I have an idea (presented below), but what can you come up with?

Answer 1

If any sort of comment-hiding is enabled, I really hope that it will be something that can be opted into on a per-site basis and that the decision of whether or not to hide comments will be made on each site's meta.

A lot of what we're talking about here seems to be specific to Stack Overflow. So maybe SO needs this, maybe it doesn't; I'm not going to take a position on that. But on the smaller sites (and mine, ELL, specifically) comments are crucial. They're used to discuss improvements to answers (which are usually then edited in) and often there's a need to ask for clarification, as answerers tweak their answering to the needs of learners. Yeah, occasionally there are a couple chatty comments. And y'know what? I don't even delete those unless they get out of hand. On occasion, in moderation, they're fun. They're usually related to the topic in some way (we like interesting puns and English jokes over on ELL). A good bit of the fun of it for me would go away if there weren't the occasional not-completely-necessary comment.

There's not a lot of moderator time involved in cleaning them up; this is the kind of thing that isn't a problem when it's scaled down to a small site. When OT comments get out of hand we delete. That doesn't happen more often than mods have time to take care of it, for us. Also, comments are really how we get to know each other as a community. There's a lot of intelligent vetting of answers in comments, which is probably what makes me trust a given user's expertise equally as much as their answers.

So this was kind of a lot of rambling that I pieced together in several sittings ;) Might come back later and clarify things or add more. But I really, really would not want to hide comments on ELL.

Answer 2

I've found a flaw in my analysis.

It turns out that I ignored most posts with zero comments. The results including those posts significantly change the picture. I'm not sure that my theory is wrong, but I need to do some more analysis in the database. There's a really good chance I was thinking too fast.

Showing comments costs a post votes.

I believe that comments are most distracting to people who are reading a Q&A page. Comments are most useful for people who are authors or editors of individual posts. It's pretty easy to measure the output of authors and editors by looking at how often they create or revise posts. It seems harder to measure the output of readers on most sites. But on sites with voting (such as ours) the output is easy to measure: readers contribute by voting (up or down) on posts.

My guess is that when readers get to the bottom of a post, they tend to forget to go back to the top of the post and vote if there are comments to read. As a compulsive reader, I can say, anecdotally, that I'm far more likely to keep reading if there's more text. It's possible that comment clutter prevents people from going back to vote on posts without extra mental effort. If so, displaying comments might cost the authors of posts reputation.

That's a good story, but how can we test it? A split test where group A sees comments as they are now and for group B all comments are hidden would probably do the trick. If people in the test group vote (up or down) more often that people in the control group, we can be pretty certain that displaying comments is a drag on our reputation-based economy.

Answer 3

We are running an obsolete and chatty comment debt.

Anecdotally,¹ very few of the extant comments on Stack Overflow (or any other Stack Exchange site) are rude, offensive, or spam. That's a testament to the rousing success of our flagging system. Over the years, the rate of rude or offensive comment flags to new comments on Stack Overflow has stabilized² to between 0.1 and 0.15%:

The absolute number of rude and offensive comments posted to the site has increased as time has gone on, but flagging of such comments seems to have kept pace easily enough. There is no particular problem with excessively rude comments on our sites. However, when I read questions on the site, there's no question that many of the comments I see are obsolete or chatty. My subjective estimate is that somewhere between a quarter and a third of all comments have been overtaken by events or are little more than social pleasantries. Here's how the flag rate on those comments has progressed:

Note that the heavy spike of activity starting in September of 2011 coincides with the introduction of several flagging-related badges on August 28 of that year. The spike is a testament to the temporary power of extrinsic motivation. It also shows that there were plenty of existing obsolete and chatty comments available for people to flag. (Note, that all data includes helpful flags only.) The drop off in flagging rate likely has more to do with people earning their badges and no longer feeling the need to flag such comments.³

In the last year or so, obsolete and chatty flagged comments have increased from about 0.1% to nearly 0.5% last month. Even so, that seems subjectively at least an order and a half of magnitude off from the actual rate such comments are being added to posts. Assuming commenting has not improved substantially, we are racking up a considerable debt of obsolete and chatty comments. Here is the number of new comments and comment flags being added each month⁴:

Why should we care?

The trouble with this theory is that it doesn't seem like a big deal if you happen to think these comments aren't such a big deal. But there are two reasons we should be concerned:

1. Broken windows

Many people see these sorts of comments as unsightly and encouraging of more of the same. The current system leaves most people no option other than flagging to clean up trivial comments. That results in a sense of powerlessness which is very unusual on a Stack Exchange site. Even if you disagree that there are broken windows, you have to agree that some people are annoyed by trivial comment clutter and (because of the deficit we are running) will never be satisfied. We have a system (comment flags) that barely scratches the surface of the problem. Flagging rude comments works. Flagging trivial comments does not.

2. Moderator time

Last month, we saw ½ million posts with a million comments. There were 188,109 total helpful flags of which 9,985 were on comments:

3722 Comment Obsolete
3618 Comment Not Constructive Or Off Topic
1179 Comment Too Chatty
 990 Comment Rude Or Offensive
 476 Comment Other

So if you don't think keeping obsolete and chatty comments visible is a big deal, moderators are wasting substantially more time dealing with trivial flags than with flags on outright rude comments. No matter how you feel about trivial comments, we are asking our moderators to clean the Augean Stables with teaspoons. Ought we not free them up to focus on important flags?

Testing a comment hiding scheme.

When testing a comment hiding scheme, count it success when the test group raises:

1. Fewer obsolete and chatty flags, and
2. No change in the rate of rude and offensive flags.

Not constructive or off topic and Other flags are ambiguous. It may be that simply hiding rather than deleting such comments will be an acceptable result. I would suggest that comments would need to be displayed for at least a day or two in order to satisfy the second condition.

---

_{1. Which is to say, in my subjective opinion. 2. There are two spikes in the first half of the graph. The first, I believe is the natural enthusiasm that comes from a new button to push. The second is likely related to the [improved flagging](https://blog.stackoverflow.com/2011/01/improved-flagging/) introduced during January of 2011. The substantial drop off in flagging during May, 2011 is likely a result of splitting comment flags into several new buckets, but I don't immediately see where (or if) that was announced. Obsolete and chatty flags begin in that month. 3. I believe that I can verify this by comparing the number of people who earned the badges to this graph. But I don't think this is a critical part of the argument. Even if badges had nothing to do with this spike, the fact that these flags have not stabilized is a red flag (so to speak). 4. I'm including _all_ helpful comment flags on this graph since comment flags seem not to have been categorized early on. Flags and deleted comments are not exported to the public data dumps. But you can verify the basic shape of the comment graph using [this query](http://data.stackexchange.com/stackoverflow/query/153354/comments-by-month#graph).}

Answer 4

I have found comments to be tremendously useful, but at the very least you could say that comments are a sacrificial metal, acting to protect the purity of questions and answers by giving people a place to post... well... comments. Hiding them would significantly reduce their effectiveness in deterring poor edits and low quality answers by the people who would be least likely to find them... new users. How can you possibly calculate the "cost" (or value) for that?

Answer 5

The risk of hiding comments?

Rarely are there comments which on their own significantly add to a post. The entirety of the comments may add content, but usually only one will shine through as the comment which caused the post to become more significant or contextual.

Hiding comments is an easy one click fix for a user, so the real risk is that a high value comment hidden would have been overlooked because the user did not know it was there.

As a user looking for value in posts, comments are often a place to look when examining the critique of a post. If there are no critiques or suggestions of value, then there is no point in examining them so I think that as long as enough are shown to entice a user to click through for the entire set then that will be enough to allow users to find the high value comments if they are hidden.

---

Measuring the risk

When being thorough, a user may want to see every comment regardless of their quality. I don't think many posts actually require the entire set of comments to have the answer be relevant.

Split test

Test for the amount of interaction done with the comments where finding them by clicking is undesirable, and upvoting them is desirable.

I think a measurement could be in the form of click through comparisons plus upvotes. Group A sees the status quo, B sees the new version (whichever version that is). Compare total points from how many times each group triggers the reputation hover for the username of a comment (+1 per username), upvotes a comment(+10), expands all comments(-5).

Whichever group has the most points indicates the most positive interaction with the comments.

Answer 6

Let's start from the basics.

1. Comments suck up moderation resources and reader time for potentially dubious value
2. Comments are really really hard to evaluate because of how they're used/how they need to be looked at
3. Despite being a complete hassle to deal with, comments play a very important role in clarifying/improving content that there is no other tool for

Because of these three issues, it is exceedingly difficult to get a clear signal from what we have now, which is why we are in this situation in the first place. Rather than trying to measure the signal we have, we should add more information to help clarify that signal, or potentially look in to changing the comment system to better reflect the different roles that comments actually play on posts.

The Problem

As Jon points out there were almost 100,000 comment flags handled by mods in a month. That's a lot of flags. As someone who's deleted several hundred comments in my corner of SE over the past month, I can tell you that comment flags are my least liked of all flags because they:

1. Need to be read in context (you have to read the entire question, answer, and the entire comment thread to determine what the heck to do as a result of the flag)
2. Have varied actions that need to be taken (you need to delete, or edit the comment, or edit the comment in to the post, or purge the whole lot of comments because it's two people bickering, etc.)
3. Sometimes are just great signposts that don't require any action but have no alternative way of being handled (for instance linking to a related non-duplicate question for a newer version to fix the same issue, etc.)

This has all been stated before, but from where I stand this means we have to focus on two separate (but related) problems:

1. Measuring if a comment has value
2. Measuring what the value of that comment is

Getting a clear signal

Right now we have three different things we can measure in regards to comments:

1. Comment upvotes
2. Comment length/content
3. Comment deletion (by author or mod or flags)
4. Comment responses (through @mentions)

As Jon stated:

People aren't perfect: I've observed that votes on short comments tend to mean "Funny". But votes on longer comments tend to mean (to take a page from Slashdot) "Insightful", "Interesting", or "Informative". Comments that we want to keep around have a combination of length and upvotes.

So the best signal we have requires a combination of factors. An upvote doesn't mean useful content, it means 'potentially useful content' that has to be correlated with other data which gives a signal that isn't clear enough to actually act on. What we need is a simple indicator of whether the comment has value in regards to what comments are supposed to be for:

• Request clarification from the author;
• Leave constructive criticism that guides the author in improving the post;
• Add relevant but minor or transient information to a post (e.g. a link to a related question, or an alert to the author that the question has been updated).

That signal does not currently exist, so we need to make one.

The easy way or the hard way

I see two ways to handle this. One is to create a weaker signal (but require less effort from users meaning more participation and a broader sample), and the other is to create a far stronger signal (but require far more effort from users reducing participation and potentially introducing sample bias).

The Easy Way

You create a one-click button next to every comment:

Participating could be limited to people over X rep, and optionally require participants to read a page summarizing what we want them to click the button for (comments that do not follow the comment guidelines). These clicks get recorded in the database somewhere, and allow us to get a clear signal of which posts users think provide the right signal according to our guidelines, and which don't.

We can view what actions users take after clicking it (do they flag? add a comment? edit the post? vote?) for further analysis, and we can look at the types of users clicking the button (heavy commenters? heavy flaggers? heavy editors? browsers? answerers? askers?) to get a better idea of what those clicks actually mean (versus what we hope it means by starting the entire process in the first place).

By using these guinea pigs volunteers, we can then test a comment-hiding algorithm on them to see how different types of actors react to the changes (do they click the 'expand comments' more? less? what types of users are using comments in which way?). This prevents any loss of signal to the community at large while still getting us some good data we can use to figure out what the heck we should do.

Regardless, this would be really analysis-intensive on the SE side since the signal would be ambiguous at best (it shares almost all the same flaws as the comment upvote feature).

The Hard Way

On the other hand, we can try to get more detail feedback categorizing posts. Using a similar volunteer method, we can give folks the ability to curate comments on their own posts by indicating what type of content is in the comment. Either creating some sort of a review queue for comments on your own posts, or having buttons next to the comments like in the other suggestion, we would want our guinea pigs volunteers to tell us if a comment is:

Clarification Constructive Criticism Minor or Transient Information Other (but good to have) Noise (Delete)

By limiting this to comments on your own posts you are likely able to get more participation (because I'm going to wager that people click through to their posts whenever a comment is added), and the information that you get will be a lot more detailed (on a more limited set of comments). This will make the analysis on the SE side far far easier, but due to the added burden for the folks actually participating, you may end up skewing the sample willing to leap through the hoops, and hurting the value of the information to more casual users.

This also has the benefit of seeing how people would be at moderating comments on their own posts, and act as a good test group to see if that sort of privilege is something that would have benefit, or would be ripe for abuse (with users deleting any comment that doesn't compliment them or other not-so-great behavior).

Either by doing the hard or the easy way, these groups could be used to test community-driven comment moderation and gather more data on how the entire process could be resolved, not only limited to hiding comments, but also to deleting them without significant moderator intervention. Either by letting folks moderate comments on their own posts, or by giving them the ability to mark bad comments and have them auto-hid/auto-deleted in the future.

Answer 7

Cost of displaying comments is the risk of obscuring valuable information. When an important comment is buried under 3-4 chatty ones, there is a high chance for the reader to miss it.

Another cost of displaying comments is that this misleads readers into thinking that these are first class citizens at Stack Exchange. Meta posts and help center articles can repeat comments are ephemeral until the hell freezes over, but tens, hundreds thousands readers seeing 3-years old chit-chat are presented with a fairly strong evidence for the opposite.

From this perspective, the more comments are hidden, the clearer is the signal sent to site visitors: "yes, we really mean it, comments are ephemeral".

Consider stressing the educational aspect of your proposal. For this to work though, there should be a consensus within SE dev team that 1) ideally, valuable information should be eventually moved from comments to posts (via editing or new posts) and 2) that additional measures are desirable to educate and incentivise site users about that.

It is important to note that focusing on educational aspect opens a door to compromise with those opposing the proposal. Think about it, if vast majority of views on "matured" questions fall at those with "compressed" comments, you can afford some relatively minor amount shown the other way.

A possible compromise could look like this: hiding comments is disabled for 3 (5, 7) days if the question gets 10 (100, 1000) views a day, or if it gets 50 (500, 5000) views a month (later is for posts with longer attention span which have lower count of views per day). Numbers here are only for example, what you end with may be different - you only need to ensure that sufficient majority (70-80-90%) of views at matured questions fall at either those without comments at all, or at those with compressed comments.

This way, you basically "trade" two risks against each other - a risk of auto-hiding valuable comments against the risk of misleading site visitors about the ephemeral role of the comments.