Let's start from the basics.
- Comments suck up moderation resources and reader time for potentially dubious value
- Comments are really really hard to evaluate because of how they're used/how they need to be looked at
- Despite being a complete hassle to deal with, comments play a very important role in clarifying/improving content that there is no other tool for
Because of these three issues, it is exceedingly difficult to get a clear signal from what we have now, which is why we are in this situation in the first place. Rather than trying to measure the signal we have, we should add more information to help clarify that signal, or potentially look in to changing the comment system to better reflect the different roles that comments actually play on posts.
As Jon points out there were almost 100,000 comment flags handled by mods in a month. That's a lot of flags. As someone who's deleted several hundred comments in my corner of SE over the past month, I can tell you that comment flags are my least liked of all flags because they:
- Need to be read in context (you have to read the entire question, answer, and the entire comment thread to determine what the heck to do as a result of the flag)
- Have varied actions that need to be taken (you need to delete, or edit the comment, or edit the comment in to the post, or purge the whole lot of comments because it's two people bickering, etc.)
- Sometimes are just great signposts that don't require any action but have no alternative way of being handled (for instance linking to a related non-duplicate question for a newer version to fix the same issue, etc.)
This has all been stated before, but from where I stand this means we have to focus on two separate (but related) problems:
- Measuring if a comment has value
- Measuring what the value of that comment is
Getting a clear signal
Right now we have three different things we can measure in regards to comments:
- Comment upvotes
- Comment length/content
- Comment deletion (by author or mod or flags)
- Comment responses (through @mentions)
As Jon stated:
People aren't perfect: I've observed that votes on short comments tend to mean "Funny". But votes on longer comments tend to mean (to take a page from Slashdot) "Insightful", "Interesting", or "Informative". Comments that we want to keep around have a combination of length and upvotes.
So the best signal we have requires a combination of factors. An upvote doesn't mean useful content, it means 'potentially useful content' that has to be correlated with other data which gives a signal that isn't clear enough to actually act on. What we need is a simple indicator of whether the comment has value in regards to what comments are supposed to be for:
- Request clarification from the author;
- Leave constructive criticism that guides the author in improving the post;
- Add relevant but minor or transient information to a post (e.g. a link to a related question, or an alert to the author that the question has been updated).
That signal does not currently exist, so we need to make one.
The easy way or the hard way
I see two ways to handle this. One is to create a weaker signal (but require less effort from users meaning more participation and a broader sample), and the other is to create a far stronger signal (but require far more effort from users reducing participation and potentially introducing sample bias).
The Easy Way
You create a one-click button next to every comment:
Participating could be limited to people over X rep, and optionally require participants to read a page summarizing what we want them to click the button for (comments that do not follow the comment guidelines). These clicks get recorded in the database somewhere, and allow us to get a clear signal of which posts users think provide the right signal according to our guidelines, and which don't.
We can view what actions users take after clicking it (do they flag? add a comment? edit the post? vote?) for further analysis, and we can look at the types of users clicking the button (heavy commenters? heavy flaggers? heavy editors? browsers? answerers? askers?) to get a better idea of what those clicks actually mean (versus what we hope it means by starting the entire process in the first place).
By using these
guinea pigs volunteers, we can then test a comment-hiding algorithm on them to see how different types of actors react to the changes (do they click the 'expand comments' more? less? what types of users are using comments in which way?). This prevents any loss of signal to the community at large while still getting us some good data we can use to figure out what the heck we should do.
Regardless, this would be really analysis-intensive on the SE side since the signal would be ambiguous at best (it shares almost all the same flaws as the comment upvote feature).
The Hard Way
On the other hand, we can try to get more detail feedback categorizing posts. Using a similar volunteer method, we can give folks the ability to curate comments on their own posts by indicating what type of content is in the comment. Either creating some sort of a review queue for comments on your own posts, or having buttons next to the comments like in the other suggestion, we would want our
guinea pigs volunteers to tell us if a comment is:
Clarification Constructive Criticism Minor or Transient Information Other (but good to have) Noise (Delete)
By limiting this to comments on your own posts you are likely able to get more participation (because I'm going to wager that people click through to their posts whenever a comment is added), and the information that you get will be a lot more detailed (on a more limited set of comments). This will make the analysis on the SE side far far easier, but due to the added burden for the folks actually participating, you may end up skewing the sample willing to leap through the hoops, and hurting the value of the information to more casual users.
This also has the benefit of seeing how people would be at moderating comments on their own posts, and act as a good test group to see if that sort of privilege is something that would have benefit, or would be ripe for abuse (with users deleting any comment that doesn't compliment them or other not-so-great behavior).
Either by doing the hard or the easy way, these groups could be used to test community-driven comment moderation and gather more data on how the entire process could be resolved, not only limited to hiding comments, but also to deleting them without significant moderator intervention. Either by letting folks moderate comments on their own posts, or by giving them the ability to mark bad comments and have them auto-hid/auto-deleted in the future.