15

We've got an analysis of 3,992 comments. How were these selected? Right now, it's not very clear at all how this entire process worked, other than comments were served up, rated, and recorded.

Some questions I have, although this is not exhaustive:

  • Did there need to be a certain number of comments?
  • For their first answers, were they subject to moderation, such as not being an answer?
  • What counts as a new user? Their first posts only?
  • Did these comments come from the live database, with deleted comments gone, or shown?

If possible, a raw dump of the data would allow us to do our own analysis on it as well.

| improve this question | |
  • "What counts as a new user? Their first posts only?" Did the analysis say anything about new users specifically? "Did there need to be a certain number of comments?" What does this mean? – Nicol Bolas Jul 10 '18 at 18:48
  • @NicolBolas According to the intro screenshot, it's presenting comments on user's first posts. Imagine you are a new user to Stack Overflow, and you've just asked a question or posted your first answer. For the comments, I'm wondering if only posts with >=X number of comments were selected. – fbueckert Jul 10 '18 at 18:51
  • 2
    I didn't take that as a statement that the comments were specifically culled from such posts, only that the person rating them should put themselves in the position of a new user. After all, the point is not to rate comments made to first posts; it's to rate comments period. – Nicol Bolas Jul 10 '18 at 18:53
  • @NicolBolas According to Shog, they're from new user posts. – fbueckert Jul 10 '18 at 19:02
  • 1
    I don't think Shog meant they were literally only new user posts, but was talking about the framing of the rating task. – Julia Silge Jul 10 '18 at 19:03
  • @JuliaSilge That makes sense when you understand the process, but it's hard for that statement to be read that way based on incomplete information. If it's not just from new users, that's fine, but all these clarifications and caveats are making it more confusing, not less. – fbueckert Jul 10 '18 at 19:07
13

These comments were not specifically made on new user's posts. The new user framing was to guide the rating task, i.e. "Imagine you are a new user..." and then see how these types of comments would be perceived.

The comments included in the rating app were undeleted comments from the live database, not already flagged/deleted comments. We built the prototype, found at what rate we were rating comments, then estimated how many comments we could rate to get multi-rater comparisons with the number of folks we have internally with domain knowledge about the community and the amount of time we wanted to ask from each person. They were all drawn randomly from a single, recent month (chosen not to be the beginning of a university semester, middle of summer, etc).

We showed all the comments in a single thread at once for context, but the comments were rated individually. A thread could be a single comment; a typical (median) number is two but they could be as long as six or so.

We have not ruled out releasing this kind of data in the future, especially once we have more of it from a broader user base, but we are not there yet. There are issues to consider around user privacy, for example.

| improve this answer | |
  • 3
    It's probably not worth anyone's time to actually do this, but I'm curious whether there would be any statistically significant difference in the results if some of these comment threads were presented with Stack Overflow design and others were presented using stylesheets from other popular sites or generic fictional sites. Even people participating in good faith may find it hard to actually "imagine [they're] a new user" if they're familiar with the environment. – SOLO Jul 10 '18 at 19:18
  • 2
    Unrelated to my previous comment, please have a conceptual +1 for your consideration of users' privacy. – SOLO Jul 10 '18 at 19:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .