41

Independent verification is a pillar of the scientific method, partly for sanity-checking and also because (and I've mentioned this multiple times before) both the cohorts used for the loop and the method are kept behind closed doors.

On an open-ended form, one would expect the average user to have multiple, sometimes competing concerns. For instance, one may flag up design and community. As such, one would expect either a sum total of events to surpass 100%, or for it to be renormalized in some way.

This is the graph presented in The Loop #2: Understanding Site Satisfaction, Summer 2019 article:

transcription below

Coded responses to "What do you find most frustrating or unappealing about using Stack Overflow?"

Unwelcoming community 10.6%
Design 9.8%
Artifact quality 9.7%
Barrier to participation 8.3%
Discovery 8.0%
Overmoderation 7.1%
Voting 5.1%
Question quality 4.2%
Timely answers 3.5%
Other 3.2%
Comments 2.2%
Onboarding 2.1%
Social friction 1.8%
Subjective content 0.8%
Mobile app/site 0.6%
Welcoming backlash 0.5%
Job quality 0.4%
Review queues 0.3%

Two things immediately stand out:

1. The dataset is incomplete

enter image description here

It should not take a genius to figure out that, even with the previous caveats (that either we are looking at a sum total of events > 100%, or renormalized), this graph is once again missing 20% of respondents.

2. Some concerns overlap

"Question quality" and "Artifact quality" are the two biggest examples of this, where somebody would have to have written an entire paragraph for a human to, objectively, without letting their bias taint the coding, identify it was one and not the other.

Due to these concerns, could we please follow the most basic scientific method steps and release an anonymized dataset for people who would be willing to cross-check the findings?

Improve this question
12
  • 12
    Yet another "The community is toxic, we need to change it" post. If this is all that SE, Inc. thinks about us, I'm not surprised they treat us like a plague. – MechMK1 Jan 23 '20 at 11:39
  • @MechMK1 this might or might not be true. The dataset will clarify that; it is perfectly plausible that this is the #1 outcome, but the fact that there is no data available puts this at the same credibility level as me writing random values in excel cells and posting the result. It is also human coding, yeah? – Sébastien Renauld Jan 23 '20 at 11:40
  • 1
    I already made this statement before, but I would say the community being unwelcoming is too broad of a statement. For example, what if my question is being closed as a dupe, even though I searched beforehand? I might feel as if those who voted my question as a dupe were being unwelcoming, as it carries an implied accusation that I did not search beforehand. The true culprit however is the less-than-perfect search functionality. Just one example of how the community is not to blame for being "unwelcoming". – MechMK1 Jan 23 '20 at 11:44
  • 2
    @MechMK1 I research my questions well, but of my past 10 questions, I received a total of 3 upvotes, and on my past 10 answers, I received a total of 4 upvotes. No-where else on Stack Exchange do I get such a poor voting ratio. Maybe my questions and answers are poor, but nobody has commented with constructive criticism either. I do find that the community was friendlier 6 years ago. So I would agree that the community on Stack Overflow (not on other Stack Exchange sites I actively participate in) is unwelcoming. So is Stack Exchange Inc. – gerrit Jan 23 '20 at 12:40
  • @gerrit I can't comment on Stack Overflow. I have not been there for years. – MechMK1 Jan 23 '20 at 12:45
  • 1
    @gerrit 3 up-votes and 4 up-votes, how many down-votes? Could you provide some examples? It seems that you're complaining that you never received enough up-votes? If that's the case then know that up-votes are not guaranteed and that should not signify how "welcoming" people are. Your comment is proof that people think voting and how "welcomed" you feel have become the same thing these days. – Script47 Jan 23 '20 at 14:47
  • 1
    @Script47 7 upvotes, 0 downvotes, on 20 posts. I believe people are voting much less than in the past or than on other sites (I had seen a SEDE query showing this, but can't find it now). There are also dramatically less answers. For examples, see my questions and answers. And of course upvoting quality content is essential and, yes, welcoming. – gerrit Jan 23 '20 at 15:11
  • 2
    @gerrit but less up-votes do NOT mean that people are more "toxic" or "unwelcoming". You are trying to find links that match your conclusion which don't fit, or, rather, haven't been proven to fit. – Script47 Jan 23 '20 at 15:18
  • @Script47 I do think that less up-votes means that the community is less welcoming. Most people appreciate to receive praise for good work, and upvotes are a form of praise (more so since we don't do "thank you" comments). For both volunteers or paid employees, if they spend months or years doing great work but no-one ever tells them so, that would be an unfriendly work environment. – gerrit Jan 23 '20 at 15:23
  • 2
    @gerrit you underestimate the laziness of some users (speaking as a ex lazy user) who simply don't vote. I noticed this behaviour in myself so I actively tried changing it (in some instances where I'd be signed out of SO and I'd sign in to simply up-vote Q&As that helped me) because I no longer was using the site to simply get answers, rather, with the end (not really ending) goal in mind; an archive of quality Q&A. Ultimately, I'd say using this as a way to gauge the friendliness of the community is deeply flawed. – Script47 Jan 23 '20 at 15:28
  • "one would expect... either a sum total of events to surpass 100%, or for it to be renormalized in some way" - surely the simplest explanation is that something above 22% of respondents left that box blank? – user56reinstatemonica8 Jan 24 '20 at 17:30
  • Personally, I think lack of complaints about the app show a huge red flag! ;-) – Scott Seidman Jan 24 '20 at 18:10
16
  1. Unwelcoming community (10.6% of responses)

My very first thought here: yes, as I a member of the community, I don't feel at all anymore that the company is welcoming my efforts.

Which I made ample clear in my written answer to the corresponding question in that survey.

So, I agree to the point of the question here: it would be very helpful to get more insights into the raw data. I really doubt that all points summarized under "unwelcoming community" are really about "the community vs new users", and not "the company vs the community".

But let me add this: thanks SE Inc. for somehow trying to keep us in the loop. I seriously mean that. I am not happy with most what comes what comes down lately, but at least, it seems you try to uphold the statements you gave earlier on regarding new feedback mechanisms.

Improve this answer
2
  • 5
    I don't really feel like I'm in the loop here. The lack of transparency in The Loop makes me feel as if it's just a way for them to choose what to address without evidence of anything being left out. – João Jan 23 '20 at 14:01
  • 1
    But we are kept in the loop about being left out. Or so. – GhostCat Jan 23 '20 at 15:18
13

It should not take a genius to figure out that, even with the previous caveats (that either we are looking at a sum total of events > 100%, or renormalized), this graph is once again missing 20% of respondents.

Not really. From the weblog article:

We manually coded 2,942 of these open-ended responses and organized responses into different themes so that we could better understand users’ frustrations with using the site.

I would interpret open-ended as being an open question. Some people might have given answers that can't be coded into any of those themes, perhaps answering that everything is fine, writing something irrelevant, or writing nothing at all. I don't find it at all strange that ≥20% of open ended answers cannot be easily coded into any category. That is still different from "other", which may be a specific complaint that is relevant but doesn't fit in any category.

Releasing the answers to free form questions is probably difficult to do without risking the anonymity of the responses, as most of them are likely unique (see also What were the complaints filed under "other" in the survey data?).

"Question quality" and "Artifact quality" are the two biggest examples of this, where somebody would have to have written an entire paragraph for a human to, objectively, without letting their bias taint the coding, identify it was one and not the other.

It appears that by artifact quality they mean answer quality. So one is complaining about poor questions, the other about poor answers. Perhaps some complain about both, but questions and answers can quite objectively be told apart.

Improve this answer
2
  • "open-ended responses" could also mean responses to open-ended questions – gnat Jan 23 '20 at 12:36
  • 1
    There's an "other" category (3.2%) which I imagine includes the responses that can't be classified. Seems most likely that something over 22% of respondents simply left that box blank. Or, it says the themes in the chart are "top themes", so maybe there's a "long tail" of niche classifiable concerns, and "other" is the completely unclassifiable (e.g. random keyboard mashing, "no", etc) – user56reinstatemonica8 Jan 24 '20 at 17:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .