Allow me to piggyback on your question since I wanted to ask about this. For the record, I'm not saying I would have done better because qualitative research is harder than it looks, but I would also like some answers to better be able to make sense of that chart. I apologize for the lack of citations, this is off the top of my head. Feel free to comment if I got something wrong.
Some questions and comments:
Inductive and deductive coding
Inductive coding means that you build codes from the data, and eventually draw themes from what these to try to make sense of what's going on. When deductive coding is used, the research starts with preset categories and fits the data into boxes. This can be helpful if you want to make a comparison with something else or you are using a framework, and probably more but it's not something I've used often. Which method was used there?
It seems plausible to think that the data was coded by hand, from Yaakov's answer here (and kuddos to Yaakov for keeping communication open with Meta):
For a number of months we have been doing this by hand (yes, a few
people have looked at many thousands of these responses, and assigned
them to one of many dozens of categories).
(Note, it's not clear here if this applies to the Site Satisfaction Survey and if machine learning was used on this data.)
I'm not sure how many people is "a few" and if they coded the data in way they could assess reliability (by coding a single item multiple times and checking for agreement), or if they just needed a few people because there was a lot of data. Was inter-rater reliability taken into account, and if so, how?
Themes
Themes are supposed to give a sense of the data by capturing the underlying idea behind it. Often, themes are an actual sentence. Currently, the themes are really vague. What does it mean to be frustrated with voting? Is one frustrated about downvotes? That they are anonymous? The lack of votes? The accumulation of votes through time or from HNQ? It's fine to categorize information but it should be looked into further to better understand what exactly is going on and point to the actual problems, then you can get to the essence of things.
What's not being said
Another important feature of qualitative research is trying to make sense of what's not being said. What's missing from the bar chart? There is no interpretation included in the post as far as I'm aware.
Sampling
According to the blog post, the data comes from a representative sample of users. I would like more explanations about the sampling method. What characteristics were used? Location? Age? Reputation? Something else? Is the sample self-selecting? As in, one may be more likely to answer a survey if they want to voice a dissatisfaction?
Bunching all responses in the chart regardless of user background
Given how privileges scale, I highly doubt that new users find review queues annoying. So it seems all responses are bunched together. It's critical to gather data from new users, but it probably should be nuanced differently than the data collected from intermediary, experienced users, moderators, etc.
To illustrate my point, the director of public Q&A wrote a great article about how it feels to get ganged up on. That it's possible each individual comment is not being rude but the sheer amount of disagreement is overwhelming. A sturdier approach would be to categorize people from what we know of their experience of the site and compare the responses across categories. Can we make better sense of some responses, such as "unwelcoming community" with other data, research, and experiences? (e.g. literature about being ganged up on? Flags for unkind, rude behavior? Rude comments classification? Interviews?)
Self-reports
Self reports were not coded taking potential bias into account. It's not "I felt unwelcome, put on the spot, like I didn't belong", it's not coded as unwelcoming atmosphere, or that there is a divide between users that needs to be fixed. Why not take some time to look at that data again and try to get to the essence of it so we can all work together towards a common goal?
Taking your audience into account
Who is the bar chart for? Is it for company use? For experienced users? For new users? If you had a bad experience with Stack Overflow, then the bar chart is comforting and validates that it's not you, it's us. If you're part of the community, well you were just classified as unwelcoming. You, the community, is unwelcoming. Can we please stop digging the hole between the community and the company?
Overall, both anonymous and registered users are highly satisfied with Stack Overflow and tell us that their favorite things about our community include finding solutions to their problems, vast access to information, and the knowledgeable people who participate.
Would the company mind giving the community a bone there? Was the positive data coded? Spending some time analyzing what the company, its volunteers, and contributors are doing properly can also be worthwhile.
