31

So. We just got another welcoming blog post. Leaving aside analysis of it, I think it's missing a critical piece of information.

We've got a tiny subset of comments that were used to compile this data. Apparently, 7.7% of them were Not Fine. That still leaves 92.3% as various levels of neutral. It's easy to lash out when you get an unwelcoming comment, so that just feeds the hostility spiral. But the vast majority are still good.

I think an important piece of data is to see how new users respond to these Fine comments. Can we get a subset of comments that meet the following criteria:

  • Posted by a user with < 10 rep, so it'd be on their own content/question.
  • In response to:
    • comments that were rated Fine
    • (optional) Or where their content was subject to curation of some type, such as editing, downvoting, or question closure.

We keep hearing we have a perception problem. What I'd like to see is if new users are feeding that unwelcoming loop. It doesn't seem balanced to have only comments from users who are trying to help. How users respond to them is also important.

Improve this question

1 Answer 1

Reset to default
13

I have no particular opinion on this request, other than to note that we'd need a LOT more data to draw any conclusions from it; this is a pretty narrow funnel. But I wanna touch on an assumption that seems to be present in both your proposal here and in a few other comments I've seen today: the current study is not excluding comments from new users. Quite the opposite!

Participants are asked to rate all comments in a thread - including those from new users. For answers, it is not unlikely that both participants in a thread will be somewhat new or inexperienced.

We gotta be really careful here when it comes to making assumptions about where problematic comments come from. Most folks don't have a lot of visibility into this, because comment moderation is both fairly rare and very opaque. Here's the kicker: among the almost 700 users contacted by moderators on Stack Overflow for rudeness over the past year, the median reputation is... 77. Yeah. Keep in mind, these aren't folks who posted one-off mildly-unwelcoming comments; they're the folks who were so persistently abusive to others that moderators had to take 'em aside for a talking to... Still, something to chew on.

As a society, we love narratives that have crisp, clear roles: the ruthless villain, the pure innocent victim, the noble hero... But as Jon noted recently:

The rescuer believes they are making the situation better, but can be enabling the victim's helplessness. Rescuers tend to be disappointed when the victim fails to appreciate their help. Rescuers fail to address the problem created by the persecutor, who often feels like they are the victim in this situation. (In fact, the roles are fluid though one is usually primary.) Often rescuers are avoiding dealing with their own struggles by focusing on other people's problems.

The drama triangle is great for making adventure serials: nothing is ever concluded and you can keep pumping out movies. It's not so great in real life. If you find yourself playing a role in this, you're best off asking if maybe next time you could do things differently - regardless of whether that role involves twirling your mustache or being tied to the railroad tracks...

This, then, is the value of doing a study: to identify situations where folks feel compelled to play a role in the drama, so that we can eliminate them. It doesn't matter if you're new to the site or a 10-year veteran; getting sucked into drama in comments is probably not what you're here for.

Improve this answer
8
  • 7
    The fact that the comments are drawn from new users as well isn't at all evident in the blog post; I think that point needs to be made much clearer. The only examples we have are where it seems like users are trying to help, devoid of all context. There are many more problems with that analysis, but what we have right now feels an awful lot like cherry picking.
    – fbueckert
    Commented Jul 10, 2018 at 17:24
  • 1
    The blog post could've probably made this a lot more clear with a lot fewer words if it'd just shown a screenshot of the UI in action, @fbueckert. None the less, it does include the complete instructions from the comment evaluator, which observe that you'll be shown the comments on a new user's post.
    – Shog9
    Commented Jul 10, 2018 at 17:26
  • 2
    It probably could have, I agree. What we have, though, is what we work with. And that's why I'm asking for more information; that front screen heavily implies it's filtering out responses new users made to those comments, since they don't address the post itself, which I bet skews the data. I don't think it's intentional, but it creates a bias.
    – fbueckert
    Commented Jul 10, 2018 at 17:32
  • 5
    Well, I just told you: it isn't. Reviewers are shown the entire comment thread. They aren't told who the comment authors are, which comments are from new users, which comments are from the author, etc.; they aren't shown the post itself, nor are there links provided - they're asked to evaluate the text of every comment in a thread in the context of the thread alone.
    – Shog9
    Commented Jul 10, 2018 at 17:34
  • 7
    See, that is important information. The examples don't make that clear at all. We really need more information to go off of. The methodology of how comments and/or posts were selected for the study would also be extremely useful.
    – fbueckert
    Commented Jul 10, 2018 at 17:36
  • I hear there's a big ol ask question button around here somewhere... ;-)
    – Shog9
    Commented Jul 10, 2018 at 17:38
  • Can't I just scope creep this to get a big 'ol data dump? :P
    – fbueckert
    Commented Jul 10, 2018 at 17:38
  • Nope. If you wanna know how the sausage is made, you gotta talk to the folks running the grinder.
    – Shog9
    Commented Jul 10, 2018 at 17:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .