Hide trivial comments

Question

Update: this particular implementation has been abandoned for now, please help us measure the cost of displaying comments instead.

If you look around the internet, you'll find that:

99.9 percent of comments are either spam, off-message or simply wasted electrons.—ithacaindy (Obviously part of the 0.1%.)

Stack Exchange does not have that problem. Thanks to flagging, our spam and offensive comments have a half-life of minutes. So the goal of this proposal is not to mess with a good thing. Instead, I propose we increase our information density by hiding comments that are not bad, but just trivial and likely obsolete.

Currently on Stack Overflow and other non-meta sites the top 5 comments by vote are displayed in chronological order. The sixth and following comments are hidden behind a link. For posts with many comments that means that the earliest comments have a strong tendency to be shown even if they are not particularly useful or friendly. For posts with 5 or fewer, the comments are a nearly permanent feature even if an edit or the passage of time makes them obsolete. To rectify the situation, I propose using:

Comment Weight

Each comment is given a weight from 0-29 based on the following criteria:

• One point for every vote up to the tenth.
• One point for each 15 characters beyond the minimum (15) capped at 9.
• 10 - age (in days) down to 0.

A comment with no votes, less than 30 characters long, and older than 10 days will have a weight of zero. If a comment of length 150 gets ten votes on the first day, it will temporarily weigh in at 29.

Comments with a weight less than 10 are hidden.¹ All remaining comments are shown in chronological order.²

Let's look at each factor individually:

Age

All comments start at a weight of 10 and decay over the next 10 days. That means that every comment is displayed for at least one day and most³ will be displayed longer. Several proposals have suggested hiding old comments. The common thread is that most comments don't matter after they have been seen by whomever they are directed to.⁴ Under the top 5 system, comments are assumed to be valuable unless there are too many. Under the weighting system, comments must demonstrate their value in order to be displayed. The age factor gives comments the time they need to gather support.

Length

Lukas Mathis conducted a survey that showed statistically longer is better when it comes to comments. While I could find no other studies to support or reject this claim, it does match my experience.⁵ Not every long comment is worth keeping around, but as the comment length approaches 150 or so, the odds are a lot better. The length factor is capped at 9 so that a commenter can't pad their work to force display. Even the longest comment must be validated by a second person upvoting it.

Score

It's very difficult to properly evaluate comments in isolation from their post. This is where users come in. Only people can tell which comments deserve pride of place. Voting allows you to decide whether or not a comment gets shown to future readers.

People aren't perfect: I've observed that votes on short comments tend to mean "Funny". But votes on longer comments tend to mean (to take a page from Slashdot) "Insightful", "Interesting", or "Informative". Comments that we want to keep around have a combination of length and upvotes.

How you can help

As a baseline, the top five scheme hides 3,059,691 comments on Stack Overflow as of November 4, 2013. There were 24,136,126 undeleted comments in total. The weight algorithm would hide 22,517,301 comments. Changing the algorithm would hide 93% of comments compared to 13% as now. I feel confident that much of that is noise, but there's bound to be some signal lost as well.

I've written a query that displays the comments that are shown by the comment weight algorithm. Please take a few minutes to explore the comments on some posts with problematic comments.⁶ Fork my query and tweak the algorithm. Write up your findings in an answer below. Let us know if you find significantly useful comments that would be lost or long comment threads that would be even noisier by this weighting algorithm.

---

1. By which I mean, they will be behind the add / show X more comments link.

2. There is a significant edge case surrounding very long, very heated comment threads. If we don't cap the number of comment somehow there will be posts with up to 55(!) comments shown. (See the top 100 which includes election posts.) The simplest solution is to keep the current caps (5 for main sites and 15 for metas). I'm certainly open to other ways of preventing comments from overwhelming answers in these cases.

3. The median comment length is 113, so half of all comments will actually be shown for at least 6 days.

4. If you need convincing, take a look at 5 random comments. This only picks short (< 100 characters) for reasons explained shortly.

5. Take a sample of 5 long comments and see for yourself.

6. For instance, back in August Toby Allen pointed out a “a whole load of snarky irrelevant and unhelpful comments”. Many of the worst of them have been deleted already, but there are still plenty of not-so-useful comments left.

Answer 1

The biggest change here is that all comments which get no upvotes will be hidden by default after 10-20 days. I am not sure this is a good idea.

20 days isn't very long at all in the total lifetime of a question. I'm not sure we've demonstrated that most comments that don't get upvoted are, on the whole, detrimental. I suspect that we get very poor vote coverage on comments, so there is not much to distinguish "This comment got 0 votes because it's noise" vs. "This comment got 0 votes because nobody bothered to upvote it yet"

I'd like to see more research into 0-score comments: how many of them are actually signal vs. noise? Then we can decide how much signal we're willing to lose in exchange for eliminating the noise.

Answer 2

A number of folks have inquired as to what the current status on this is, so here's an update:

Our internal discussions and analysis have led at least some of us to conclude that the proposal may do more harm than good.

While the selection method Jon proposed, particularly the length input, is really clever, when we reviewed an (admittedly small) sample of questions where it would hide comments, it's pretty safe to say the following:

It essentially hides everything but a small percent that seem to have a very strong quality indicator.

This led a couple of us to conclude that:

It clears out a bunch of noise, but takes a fair amount of signal with it.

In reviewing some posts where the comments would be hidden, there were a lot of examples where the comments, while far from awesome, seemed to add some value to a reader by being default visible, by adding information, clarifying a point, or simply asking a question so no one else needs to. Some examples:

• Requests for clarification or details
• Comments that add color, like the pros and cons of a proposed solution
• Comments that explain why an answer that sounds good won't actually work
• Requests for additional code, and the response explaining why the OP can't share it (which prevents others from asking the same thing).
• Feedback that the suggested solution didn't work (conveying to others that it looks good, but isn't actually effective
• You actually need to start here and get more information before asking this question

Note that some of the above are not ideal use of comments. Some of the places it felt harmful to hide comments, they were being slightly misused, because an edit would be more appropriate. But the key is that edits weren't happening, so the comment being visible to a visitor who'd otherwise miss the warning, etc. is better than it being hidden.

Another observation:

There appears to be more low-hanging-fruit for comment hiding on Questions than answers.

• Answer comments were more often signal.
• Plus the noise harm is much bigger on questions, as comments there could be pushing the top answer off the page, (as opposed to bumping a lower-ranked answer for questions).

We haven't given up on this, as there's a lot of strong of internal belief that we can do much better here, but this particular implementation has left us fairly split on whether its a net win or loss.

We'll keep you posted.

Answer 3

Thank you all for the feedback! I'd like to propose a few tweaks to cover some of the concerns that have been raised so far:

We need Top N

If you look at a post with an insane number of comments it becomes clear that we need some sort of limiting factor. The top 5 (15 on meta) has worked out pretty well in terms of showing the right number of comments though I'd argue it doesn't always pick the best comments. voretaq7 wrote a query that orders comments by weight. Look at the query with some long comments threads and consider how sad you'd be to have to click a link to see the 6^th and following comments.

Grace Period

Something that bugs me about the top N system is that new comments aren't displayed and don't have much chance of getting displayed. It's an excellent example of wealth condensation. That's why I designed the weight system to show every comment for at least one day. But after I wrote up the question, I started thinking about the "day in court" concept. If grace period is tied to the other parts of the weight system, some comments will be seen for a day and others for 9 days before their fate is decided. That's not exactly fair. (Though being fair isn't a primary goal of this feature.)

Including age also makes the implementation a bit more complex than it should be. So instead of making age part of the weight calculation, I propose giving every comment a grace period of 7 days. That should be long enough for good comments to gather the votes they need to be permanently eligible to be shown. So the weight factors become just upvotes and length.

To be clear, the grace period should override the top N criteria. If a comment is less than a week old, it will be shown no matter how many comments are in a thread. That means there will be some huge comment blocks at times, but they won't last more than a week.

Don't hide just one comment

Singleton comments are a little bit different than longer comment blocks. One of the goals of the proposal is to increase information density. This line is very nearly devoid of content if there is only one comment:

add / show 1 more comments

In terms of density, it would be far better to show this instead:

Try ulimit -s. If that is not unlimited, set the stack segment size with ulimit -s unlimited. – Jon Ericson Mar 15 at 0:02

Or even:

Welcome to Stack Overflow and thank you for the suggestions! – Jon Ericson Apr 18 at 15:35

There's no need to substitute a nearly content-free link for a potentially useful comment. If we suspect that even half of all comments are useful, it probably doesn't make sense to hide twin comments either. As Esoteric Screen Name demonstrated we'd still be hiding a lot if we set the threshold for invoking the hiding algorithm to 2 comments on a post.

Answer 4

A little SQL fiddling on the query - This one shows the computed weight and factors for comments (sorted by weight, then creation date).

The top N rows in the results are what will be shown (where N=5 for main sites, 15 for meta) - I'm using Creation Date as the tie breaker value when multiple comments have the same Weight. Everything else would be behind the "view/add more comments" link.

---

TODO (for someone feeling ambitious): Play with tie-breaker algorithms.
It's possible that Score would be a better tie-breaker than Creation Date.
A modified "weight" (just summing score, length, age) may also work well.

Answer 5

There seem to be two goals here: better automatic identification of comments which can be hidden, and reducing the incidence of immediately visible and lengthy comment chains.

As David Fullerton and McNab have pointed out, hiding potentially helpful information without just cause is detrimental, and would frustrate the first goal. Also, users are lazy (this one certainly is), and clicking the "show comments" link is just so much effort. Thus, I think that hiding 93% of comments is too aggressive.

I suggest modifying the algorithm to hide comments as dictated, but also only when the hosting post has at least X number of comments (requiring # comments > X). For example, a one month old, 20 character comment with no upvotes scores 0 and would be hidden. But, if it's the only comment on a post, there's no reason at all not to show it.

Here is a query showing the effect of hiding comments only when the number of comments > comment threshold. Results:

Comment threshold :: Comments hidden

• 0 :: 22,517,301
• 1 :: 19,708,413
• 2 :: 15,785,016
• 3 :: 12,406,656
• 4 :: 9,507,576
• 5 :: 7,265,864

Using the proposed point based algorithm and requiring a minimum of 6 comments on a post (a distant parallel to the current display-top-5-voted approach) still more than doubles the number of hidden comments on SO, and runs a much smaller risk of hiding useful content.

For reference, here's a count of the number of posts with a given number of comments on SO (included in the SEDE query):

num_comments post_count 
------------ ---------- 
0            8366059    
1            2997669    
2            2093611    
3            1207316    
4            777295     
5            481583     
6            318893     
7            205312     
8            135030     
9            89170      
10           59728 
...

Answer 6

When to hide comments? In my opinion...

Never hide all comments

Commenting is a way to balance out posts which are either inaccurate or disputed. When an issue is raised, then the post's votes may be affected by the comment's content because it raises an unaddressed issue. However, I do not believe the mere presence of a comment negatively affects a post.

Always show the first two

Always show the first two because it sets the tone for the "thread" which follows. I know Stack Exchange is not a forum, but the comments sure seem to follow that path where the post( question or answer ) which is the parent to the comments is the start of the thread.

Apply your metric to any past the first two to get another 2 as a summary

Of 19 points, 10(max) for votes, 9(max) for content length - I am not sold on the days metric. Take the set of comments with points > 1, order by total votes then by date posted descending (oldest first), and then display another 2 of those.

Hide the rest

Hide every thing else with the "add/show more comments" link.

Summary

Take the first 2, add in 2 qualified comments, and hide the rest.

Answer 7

I can't quite write a query for this (ot enough SQL-fu), however it shouldn't be hard to implement:

Why not hide long conversations by default? I've seen these happening, and they seem to generate comment votes, too, and clutter up the answer area.

If there are more than 3 comments from 2 users, replying to each other, hide them all (except for the more upvoted ones maybe)

---

Why not give comment replies in general more weight/points? It seems like it's pretty common for them to contain useful info after being asked for it. Especially replies by the OP, except maybe excluding replies that start with "Thanks" and/or contain the words "updated"/"edited". Ideally, these should be incorporated into the post, but we can't automate that.