If 95%+ of comments have been rated as "fine", does the site deserve its reputation of "unwelcoming"? Do we still need to focus on it?

Question

Quick review

There's been much ado over the last 5 years or so about the SE network being "unwelcoming".

From the summer of love to the code of conduct, from the welcome wagon to the instantaneous delisting of IPS from the HNQ because of a complaint on Twitter, there's been an enormous and ongoing effort from Stack Exchange to make the site more welcoming.

And there's been an equally enormous amount of drama on MSE (and other Metas) about these efforts.

Commentary on comments

Most recently, Stack Exchange has focused on comments as the principal source of the perceived unwelcoming atmosphere.

And they ran a well-constructed experiment to test this hypothesis. And now the results are in.

Net-net, more than 95% of all comments are considered totally fine. Less than 5% are considered "unwelcoming", and a negligible number are considered "abusive"¹. There is a high-level of inter-user agreement on these ratings. Agreement is unaffected, mostly, by reputation.

Welcome, one and all

Which makes me wonder: is there a problem to be solved here in the first place? Is Stack Exchange, despite all the drama, brouhaha, and blog posts, not actually unwelcoming after all?

Or maybe the recent focus on comments was the wrong target to aim for? If that's the case, we'd have to look somewhere else. Are we going to analyze, say, downvotes, closevotes², flags, next?

I have my own views on these matters, but I'd like to hear yours. I'd particularly like answers representing different classes of users, in particular from SE employees, from CMs, from non-CM mods, and from regular users (both high-rep and new).

---

¹ _{The linked blog presents these numbers with a lot more clarity, context, and detail. But for the sake of this MSE post, I needed to condense the results into a single number. I did that by taking the average rating weighted by the number of raters, to approximate the score the comments would get if you plucked a random guy off the street and didn't know if he were an SE employee, a moderator, a high-rep user, a new user, someone who had never heard of SE before, etc.}

² _{Actually, SO did run an experiment aimed at analyzing closures, Project Reduplication of Deduplication. But while the meta community saw this as a refreshing breath of air on investing in AI-powered tools to help automate some of the grunt work of curation, it turned out that SE's interest in the project was to help identify false duplicates.
This suggests that SE thinks closures might be contributing to the perception of being unwelcoming, and are looking for ways to mitigate that.}

Answer 1

I was once at a festival with some friends. Somebody camping across the road from us was being a disruptive jerk, and I commented negatively about it to a friend. The friend said to me: "there are 10,000 people here. If only 1% of the population is jerks, that's still 100 people."

Perspective is everything.

We don't tend to notice the vast majority of innocuous, even friendly interactions. We notice right away when somebody is being a jerk. How often does that happen? Not very. Does it make more of an impact when it does? You bet!

If 5% of comments are problematic in some way, that's one in twenty. How many SE comments does somebody typically encounter in a day? There are currently 15 on this page alone. So maybe a casual visitor won't always see a problem comment, but if it happens every second or third visit, is that something to be concerned about? Because it doesn't take many bad comments to get there.

Now, I disagree with how SE has handled some of these problems (sometimes quite strongly), and I do think some people go looking for opportunities to be offended, but I also know, from direct experience, that some of our sites do have problems with comments. What we should (and shouldn't) do about that is far from clear, but to speak to the question you ask: yes, I think 5% bad is enough to pay attention to.

Answer 2

SE is much less hostile than most of the sites and forums I've seen. So I'd certainly argue that the reputation as being unwelcoming is undeserved in this regard*. But the problem is that "we're better than the average site on the internet" is still a very low bar to meet. And we certainly should aim higher than that.

5% is actually higher than I would have expected. Though I'd argue that is because I overestimated the amount of simple noise in comments compared to unwelcoming content. You don't need that much hostile content as a percentage of total comments to create a seriously hostile atmosphere. You usually remember the negative stuff, while the other neutral comments get lost in the noise. For a new user, it only takes a single bad comment to give a really negative first impression. And many users might not try again after that.

A single snarky or hostile comment is also likely to create a response, and the tone usually doesn't improve in those cases, it only gets worse. Stopping this kind of deteriorating conversations earlier is usually a good thing.

^{* there are other aspects of SE sites that are often perceived as hostile. Getting downvoted and getting your questions closed isn't the same thing, but it certainly is perceived as somewhat hostile by many new users.}

Answer 3

Short answer: yes. We still need to think about and work on this.

Statistics are great, and I am a firm believer in data-driven behavior for most things, but numbers don't work so well when it comes to feelings.

If some new person comes to Stack Overflow and their own perception is that they haven't been welcomed, then the site is unwelcoming, regardless of how well or poorly the interactions they were involved in might be quantitatively scored.

As Abraham Lincoln would wouldn't* tell you here, there will always be some people who individually think the site/network is unwelcoming. It becomes a problem if the number of those people grows to a certain size, or those people start convincing others about how the site feels.

An alternative, slightly more concrete argument: it doesn't take a huge percentage of "bad" comments to make a site feel unwelcoming. If most of the content someone sees is unremarkable, but there's one comment that makes a new user feel uncomfortable, that's the one they'll focus on; it's human nature, not a flaw of the user.

It's just not memorable when things are boring or quietly work the way they're supposed to. It takes something unusual (either good or bad, but "bad" seems to take less work) to make a strong impression. This is why journalists talk about "dog bites man" not being particularly newsworthy, while "man bites dog" is.

Pithy wrap-up about negative perceptions: it's not always our fault, but it is always our problem.

*: Actually, it turns out the "you can please some/all of the people all/some of the time" attribution is disputed on Wikiquote.

Answer 4

It really doesn't take all that many users being mean/rude to make a person not feel welcome. I don't have the StackExchange App on my phone anymore because the comments just two users kept leaving me. One or two users can be all it takes to make someone feel harassed.

At the same time, I have raised quite a few comment flags, and in my experience, most of the rude comments are made by a very small number of users who leave lots of rude comments. Focusing on those users and perhaps removing their ability to comment on other people's posts would greatly help the situation.

Here is the other thing, outright rudeness/bigotry/racism gets dealt with right away, constant sniping and putdowns can take months before anything other than comment deletion happens.

In summary, I think that 5% of comments being unwelcoming/rude/non-constructive is a problem, but I also object to any attempts to paint the community as a whole as unwelcoming.

Answer 5

Which makes me wonder: is there a problem to be solved here in the first place? Is Stack Exchange, despite all the drama, brouhaha, and blog posts, not actually unwelcoming after all?

I'm going to be hugely unpopular and say Yes, there is still a problem. And yes, Stack Exchange probably still deserves its reputation as being unfriendly.

The following comments have all been deleted because they were either flagged by me or by someone else.

The following comment on EL&U was addressed to me after I had defended an answer as being perfectly legitimate even if the author quoted some Google statistics.

This was addressed to a new contributor who had not capitalised the word "english" in their question. Note the diamond.

This was a comment left at an answer of mine on Meta, some of you might have seen this one.

This is a comment addressed to a new contributor, which I flagged earlier today

What kind of impression do these comments leave on visitors, users, and new contributors? Who is the real victim? The recipient or the site's reputation and good name? How many users just shrugged their shoulders, and brushed the comments to one side? And by the way these comments were posted months after the CoC.

N.B. A “GP” is British for general practitioner, in American it would be “family doctor”.

Is anyone going to say there isn't a problem?

Answer 6

All of the comments in the study are from Stack Overflow, but most of the unwelcoming comments I've come across are on other sites. I understand Stack Overflow is the behemoth in the room, but I wonder if the smaller sites have more impact on the network's reputation than might be expected by simply looking at traffic.

In my experience, developers talking to developers have a higher tolerance for directness that seems rude to people working in less technical fields. When you have developers talking to, for example, an expert woodworker, there might be more friction. There are also versions of Stack Overflow available in languages other than English, which could also be a factor in reducing comments perceived to be unwelcoming. I have often seen innocent remarks that seem perfectly fine to me be very hurtful to someone else because of a language or culture difference.

In short, I'm questioning extrapolating the results to the entire network. I think that it is easier for cliques (which to me are the definition of unwelcoming) to form on smaller sites and there is more room for miscommunication when you're talking about topics where you can't just compile the answer and run it to see if it is correct. People solving a coding problem behave a bit differently than people answering a question like Is the conspiracy theory of “Cultural Marxism” inherently tied to antisemitism? (a question on the Politics site).

Answer 7

I'd wager that whether 95% seems low or high (or just right) depends on your prior assumptions. Perhaps the most important sentence of the blog post reads:

[We] as employees learned that we don’t always perceive problems in the same way as other members of our community.

Employees as a group rated more comments as unwelcoming or abusive than users. Based on our priors, the user ratings seem somewhat more optimistic about the current state of comment friendliness. Perhaps the reason we rated comments as less welcoming than other groups is because we were primed to see negative comments because it's a common complaint about the site. It is one factor in why Stack Overflow's growth has leveled off and is a problem we've been eager to solve for years.

But what is an acceptable level of unwelcoming comments? Perhaps you don't see any particular problem with the tone of the network. In that case, the current level is fine. I guess. Conversely, if you are concerned that comments are too harsh, the current level is too high. My guess is that most people who spend time on the site would estimate that somewhere between 90-99% of comments are "fine". Generally we all agree Stack Exchange comments tend to be better than comments on other sites. So finding out that 95% of comments are fine won't change our prior understanding of the "welcomingness" of the site. Neither would 91.8%, for that matter.

So what's an objectively defined level of too many unwelcoming comments? I think the answer very much depends on whether you care about the network being a permanent resource or if you are willing for the network to shut down in a few decades. This might seem overly dramatic, but I encourage you to walk down this mental path with me. (And please excuse me for focusing on Stack Overflow in the beginning. I'll get to the rest of the network near the end.)

Usenet: a cautionary tale

I got interested in newsgroups when I started programming full time in 1999. I distinctly remember an article by Jon Udell that opened my eyes to the possibility of what we used to call groupware. It was already a well-matured system that let ordinary programmers help each other. It was also "overwhelmed with spam, smut and nonsense". Fortunately, some groups (notably the the Big 8) were relatively free of these distractions. While some groups relied on moderation, many thrived simply by using social pressure. Obnoxious users would be reprimanded by Usenet grognards. One strict rule was to ignore certain troublemakers (trolls). There were essentially no other controls to keep quality high and noise low.

Stack Overflow essentially made Usenet irrelevant for finding answers to programming questions. There are a number of reasons:

• Q&A is a better format for knowledge propagation than branching discussions.
• Stack Overflow content has always been favored by search engines.
• The reputation system encouraged enlightened self-interest.
• Features like voting and question closing surfaced useful content.

But also Stack Overflow had not yet developed a reputation for harshness. Since there weren't any rules in the beginning except those created by the system, it felt safer for my generation of programmers than Usenet with its killfiles and exhortations not to feed trolls. For people like me it was an enormous opportunity to forge new rules and a new way of interaction.

So Usenet lost out to a superior product and also to a fresh culture. After 10+ years, that culture is not so fresh anymore. It's no longer possible for a handful of users to shape the way people interact. While I think we've mostly ended up in a good place, our interactions with each other are far from perfect. For the moment, I don't see a viable alternative to Stack Overflow. However, when it does arrive, I'm certain the opportunity to start over will be attractive to many existing users.

Artificial societies

We might pick 0 unwelcoming comments as an objective goal. The trouble with this zero-tolerance approach is that it leads to shifting standards. This is how we end up with a student arrested for bringing a clock to school. It's also not entirely clear how much of the problem is related to public perception and how much is related to harsh comments that still exist on the site. It could be that we could eliminate unwelcome comments and still be seen as too harsh.

On the other hand, it's obvious that there is some level at which unwelcoming comments discourage users from using the site and actively drive them away. Everybody has their own level of tolerance for unwelcoming behavior, so we need to look at the effects on the society has a whole. It's not the sort of thing we can solve with a vote or using our gut.

To illustrate the current situation, I created an agent-based model of how users might respond to a potentially unwelcoming environment. I built the model based on Shamus Young's Philosophy of Moderation. He postulates that there are three types of users:

1. Unshakable kind people,
2. Unrepentantly harsh people and
3. Normal folks who adjust to the current tone they see around them.

I built a very simple model that you can try out. (Click the "Model Info" arrow for a detailed description and "NetLogo Code" for the source code.) It assumes that kind people will always raise the level of discourse, but will leave if there are too many unwelcoming comments. Harsh people influence the environment by increasing unwelcoming comments and normal users adapt to the average level of their neighborhood. The result is an very stable situation that seems entirely sustainable for a long period of time. But every now and then a user with low tolerance for unwelcome comments will leave the site and when enough have, it will start a cascading failure case:

This is probably a good time to say I started writing the model late last week and this is the first time I'm sharing it with anyone. This by no means motivated the Welcome Wagon project and is my own idiosyncratic way of looking at the world. But I think most of us in the company agree that our public sites, and Stack Overflow in particular, are at risk of cultural calcification. If we become too insular, the next generation of programmers could very well pass us by.

The critical number in the model is the level of unwelcoming comments that will cause people to start to leave. I estimated that people who have high standards for tone will leave if 20% or more of the comments they see are snarky. That can happen from time to time even if 90% or more of comments globally are just fine. So I'm not really any closer to knowing what the "right" level is, but we can't necessarily feel safe at the current level.

Chat: a cautionary tale

This pattern usually takes a long time to develop in the model. Without examples, it's difficult to validate the model. We've had a few sites fail dramatically in their first few weeks because of increasingly crase content. The cases I recall involved "Sexuality" sites that started off poorly and quickly descended from there. We finally got a site that works for this topic by expanding to Interpersonal Skills in general and having a team of hardworking users uphold certain standards. The model suggests that if these users left, it wouldn't take long for the site to fall into 4chan territory.

You might intuit that it's a natural consequence of the topic matter. But we've see the same pattern several chat rooms on the network, including several language-specific rooms on Stack Overflow. Everytime this comes up, we discover that users serious about civil discourse have given up on the room and the users who remain believe it's fine for a room to have a culture of coarse jokes and irreverent commentary. In other words, my simple model captures a common failure case in chat. Since chat is more fluid than onsite comments, it's not surprising we'd see these problems develop more quickly in that medium.

What about content?

As I thought about the model for unwelcoming comments, it occured to me that it would work just as well to model post quality. The dynamics seem similar in that some people have high standards, others have low or no standards, and the majority will tend to the current level of quality they see around them. What happens in the people with high standards start to leave? Well, I expect quality will fall off a cliff at some point.

So why shouldn't that be the priority? Well, the first reason is that it has been since the very first question was asked on the site. If you look at the tools we have for controlling content, they are quite varied: voting, closing, locking, protecting, editing, deleting, reviewing and quality filters. At least one of these tools ought to address quality problems when they crop up. Our tools for dealing with unwelcome comments (flags, deletion and suspension) are less flexible.

My second reason is more nuanced: quality is contextual. When I read complaints about quality on the site, it's not uncommon for people to focus not on obviously junk content, but on posts that have more subtle flaws. Maybe the code should be checking for system call errors or avoid a certain security flaw or stop using global variables for everything. It's not so much that answers are wrong as that they fail to exemplify excellence in the craft of programming. It might be acceptable to have such code in a throw-away script, but it shouldn't be used in a public answer where unsuspecting new programmers might learn bad habits.

One of the things we've learned through user research is that programmers read more than just the top answer to a question. This is good because it's not uncommon to see the top two answers of popular questions be 1) an ultra-pragmatic answer that gives a solution and 2) a detailed answer explaining the nuances readers might come across. (Randomly chosen example: How do I redirect to another webpage?) It might be that the best answer for a particular situation is the quick and dirty one. But there's also plenty of room to add another detailed answer that demonstrates the best we have to offer.

There's one more reason I think we should be working on more welcoming comments: we haven't given up on quality. Right now, we are testing custom question lists, which will help people find questions to answer. Also, we will be testing the next iteration of the ask a question wizard soon. And there are a few more projects in the early stages of discovery which ought to help users produce better content.

Everyone codes now

Our annual survey reported:

Each month, about 50 million people visit Stack Overflow to learn, share, and build their careers. We estimate that 21 million of these people are professional developers and university-level students.

So who are the ~29 million people who visit Stack Overflow and are not developers? Some of them are astronomers who use R, journalists who use Git and Community Managers who use NetLogo to get their jobs done. Every indication is that there will be more non-programmers in the future. I think there are two possible consequences:

1. Code that non-programmers use can be less concerned about best practices.
2. Non-programmers will need help from programmers to get applications that meet their needs.

When I worked at JPL, I was responsible for processing spectrometer data into useful atmospheric science products. The scientists wrote their algorithms in FORTRAN and provided executables that we ran on global observations. For a few months after launch, everything seemed to be going well. But we started noticing that the science algorithms were taking longer and longer to finish. After some investigation, we discovered the executables were pulling in all the rows in the database into memory and then filtering down to the relevant dataset. Not only was that a lot of unneeded I/O overhead, but the algorithm used to select the data was something like O(n²). So as we added more data, processing took exponentially longer.

Since it was too late to change the science software, we created a hack: every discrete dataset would go into its own schema and we'd pass the connection string for that subset schema to the science executables. Eventually, we built an entire system around that concept that created schemas from the complete dataset as needed. I would have prefered to fix the queries so they had a where clause. But my job was to make everything work, so that's what I did.

It's far too late to say that Stack Overflow is for programmers; we already have a majority of non-developers using the content on this site. So the question is how do we accommodate those who are new to the mysteries of code. Yes, we need to be strict about pure junk. (But please don't use comments.) However when confronted with code that, say, will never scale properly, we ought to kindly and helpfully point that out. This is, afterall, a teaching opportunity.

The diversity of Stack Exchange

Ok, let's get real. The immediate reason we started the Welcome Wagon is that our annual survey annually reveals we don't have a very diverse user base on Stack Overflow. There are good reasons to worry about that even if you don't care about bad PR. Like most companies, we do care about getting good press, so this is likely to be one of our priorities for as long as it is seen as a problem.

We have a lot more diversity sitting just outside programmer Jerusalem. The network is still male-dominated, of course:

But the sites themselves host an incredible variety of topics beyond programming. It demonstrates that the Q&A format can be welcoming to a variety of people. I'm incredibly proud to be a part of these communities and we ought to do what we can to avoid the sort of catastrophic collapse that my model suggests is possible for any of our sites.

Answer 8

There was a little girl,
Who had a little curl,
Right in the middle of her forehead.
When she was good,
She was very good indeed,
But when she was bad she was horrid.

Henry Wadsworth Longfellow

My grandmother used to recite poems, that one was one of her favorites.

It comes to mind because I think that's what we often see on the network here. When we're good we're very good indeed, but when we're bad we're horrid. For the most part folks get along and behave themselves, but when things go off the rails... well... it's horrid.

Yes we're better than most places on the net when it comes to deleting the horrid, but "better" isn't always as good as it could be, and deletion usually occurs quietly after the damage is already done. It's easy to delete things and pretend the horrid didn't happen, but when it's directed at you, you usually see it before the rest of the community, you see it before the mods. Someone writes the horrid, you read the horrid, and then moderation wipes it away.

I've been around long enough to notice that this thing seems somewhat cyclical. We go on a campaign of nice, or welcome, or what have you, things improve enough so it's no longer a priority, and then we backslide till enough people complain, and it becomes a priority again.

If we want to break the cycle the cost is likely going to be sustained attention to these things. We don't get to say "well we're better than most" or "it's good enough for now" and take a break.

Answer 9

Your question is "Is Stack Exchange not actually unwelcoming after all?" My answer is No. Stack Exchange in general is not unwelcoming. On the contrary, I have participated in a few different exchanges over the years, and I find most sites welcoming. Only one exchange made me feel completely unwelcome, and that one experience did dampen my desires to look at other exchanges I might enjoy. And the issue with that one experience was not the answer I got; it was entirely the comments. These were comments from the community moderators! And judging from one of the other responses to your post, I'm not the only one who has experienced an issue with that particular exchange.

In my opinion, the employees at Stack Exchange are right on target: Comments need more attention. Not only should SE be focusing on comments, extra scrutiny should be placed on the community moderators because their comments have more weight than comments from some random user.

My only questions is can an exchange be helpful once the mods have formed their clique?

Answer 10

There's been much ado over the last 5 years or so about the SE network being "unwelcoming".

I can't talk from an informed standpoint about other sites on the Stack Exchange network, but, in the main, Stack Overflow isn't unwelcoming. It's simply rules-based and insensitive.

Insensitive to the point of abrasiveness.

New users think they're being criticised when they're being corrected without even a scintilla of human sensitivity.

It's only when you get past 5k rep (or so) and you find yourself still being abruptly corrected and having your questions shut down, frozen out, ignored and dismissed just as insensitively that you realise that the same abrasiveness is applied to all members no matter how new or old they are.

Then you fully comprehend that Stack Overflow is not "unwelcoming" - it's just following "house rules" and applying them to everyone like an EQ-devoid robot, without prejudice, fear or favour.

Answer 11

I came across this discussion while considering this exact issue. I believe SE does have a bit of a problem with seeming to be unwelcoming. For me the key point was hit on in the original question here:

Or maybe the recent focus on comments was the wrong target to aim for? If that's the case, we'd have to look somewhere else. Are we going to analyze, say, downvotes, closevotes², flags, next?

I'm coming from the perspective of someone relatively new to SE and focusing mostly on the English language & Usage and English Language Learners stacks. I really like the standard of rigor that users apply and the heart put into researching answers, and I've found most of the comments to be helpful and useful. That's why I joined the site and began participating, but I was quickly put off by behavior from more senior members.

What I have found unwelcoming has been the close voting of "imperfect questions" and downvoting of legitimate answers to questions deemed unfit for the community. I understand that many of those on the site are scholars and experts who expect a certain level of self-study, and there are questions from lazy students which deserve to be refused without some research, but my feeling is that it goes too far.

Closing questions because they are not rigourous enough, or don't have an accompanying essay explaining how their research left them unclear on some fine point seems harsh to me. Thus far I haven't asked a question which has been voted closed, so this isn't a personal reaction--at least not on that score. The downvoting of fair answers to imperfect questions is another matter.

My suggestions are these:

1. Users should reserve downvoting for answers that give incorrect answers.

If it's incomplete, make a comment on the answer or give a more complete answer. Don't upvote. Instead, upvote the better answers. The cream will rise to the top.

2. Users should reserve the option to close answers for duplicates, wrong category, incomprehensible, or [edit] clearly of no value to the community.

If truly a duplicate, put a link to to the older thread when closing. If it's in the wrong category, have the thread moved. If it's comprehensible but poorly worded, edit the question or ask for clarification. (edit: I understand much of this is already being done.)

If a question has the appearance of an academic inquiry that lacks effort, flag it as such and ask them for what research they've done (like what happens now), but if it's a simple question that can be easily answered either leave it for someone who is willing to handle such questions or just answer it. Trying to discuss why it is they don't understand something is usually less productive than giving a good explanation.

[edit] The standard by which questions are accepted can be broadened without compromising the quality of answers on the site.

And if there is a thread where someone asks a question that may not really merit an answer, but some kind soul does put in a decent effort, let the answer be unless it's actually unhelpful to the asker. If it's just a personal feeling the question isn't good enough, I think downvoting good questions is antithetical to the purpose of the site. At the very least it is unwelcoming.

But, I'm new. Maybe I don't get how closing questions that can actually be answered or downvoting fair answers to such questions helps things. I'm just saying that it does feel unwelcoming to me and I suspect it has the same effect on others.