Why is "The Loop" survey asking about race, age, and gender?

Question

While I understand that "The Loop" is not a research instrument in the same line as what a social scientist would develop, it is still a survey instrument and reflects upon us all (i.e., poorly designed surveys actually lead to fewer people taking them overall). As such, most surveys follow the American Association for Public Opinion Research (AAPOR) Best Practices for Survey Research.

Looking at the survey, I really don't understand why demographic information (i.e., race, age, and gender) are being asked since it appears that the clear goal is to get an initial impression of "how satisfied are you with your experience using Stack Overflow?" While I understand that a lot of people will include these by default, unless it is relevant, it's generally taught in survey methods courses to avoid asking the questions.

In this case it appears that your sample frame is effectively all developers in which case demographics really aren't needed​ for this particular question. In scientific research, demographics are only requested if there is evidence that a demographic attribute will lead to a significant difference in responses (ex., "Does a respondent's age influence the amount of money they save for retirement?"). Otherwise, demographic questions are left off to avoid respondent fatigue and because paper and ink is expensive!¹

Another reason why this is problematic is that the question is asking about "racial background" but the survey appears to be directed at an international community. Needless to say that from that standpoint, the demographic information looses all of its value as a data point since the respondents don't belong to the same cultural background, i.e., someone from Japan has a vastly different perspective from someone from China even though they may both respond "East Asian." This actually reflects very poorly upon Stack Overflow since it represents an extremely narrow perspective and​ understanding of culture.

If the "The Loop" survey is intended to be a preliminary instrument, then a better way of approaching things would have been to code the qualitative data and develop a new survey instrument (or panel interview more likely) from that.

As such, per the title, why is "The Loop" survey asking about race, age, and gender?

---

1. Obviously that last point isn't relevant online, but you may or may not be surprised at how much work is done to limit the length of a survey to save on printing costs.

Answer 1

Because that is what Corporate actually cares about. Or at least race & gender. Probably age to a lesser degree. The "gender" part has been emphasized recently via the "pronoun problem", but both race & gender have entered into the general world out in so many flavors. Call it "affirmative action". Call it "inclusive". Call it whatever you want.

My personal take on all of this: Programming (and nearly all the SE network of Q&A, but especially the programming and other technical sites) should be a race, age & gender-agnostic field. For better or worse, that is not always the case, but that should be the goal, not to make it "better" for a historically disaffected group but simply "great" for "everyone".

The question becomes whether to somehow proactively figure out how to "solve" that problem by asking who your users are (which appears to be the case with this survey) and somehow use that information to make the site better (if group 'a' seems happier than group 'b' then figure out how to make things better for 'b' - a noble goal if you can actually do anything about it) or to simply produce a product that has no inherent bias for/against any race, gender or age, where everyone is welcome and nobody cares if you are male, female, non-binary, young, old, white, black, etc.

SE historically has been a site where you could ask a programming question and (unless you wanted to let people know by way of your username or an actual photo in your profile) nobody would know or care whether you were male, female, non-binary, young, old, white, black, etc. Because C, PHP, Python, Java, etc. do not give different error messages for women and HTML/CSS produces different output in different browsers but not dependent on the age of the user.

Now if you're trying to sell a company to some big "words are all that matter, not deeds" corporation, then maybe this all matters. If you are trying to simply provide a place where people can ask technical questions and get answers, it doesn't matter AT ALL. Maybe a different system like Codidact (I am one of many people involved in developing Codidact, and there are many other open source Q&A systems out there) will do a better job of that, focusing on the users and not on political correctness.

But I digress.

Answer 2

Race, age and gender are just a part of the larger identity politics that has been seeping into tech.

For a larger context, I refer you to the social psychologist Jonathan Haidt:

Q: Are these problems graduating with Gen Z to the workplace?

H: While we were writing the book, I was beginning to hear anecdotes from business people, but we didn't have enough to go on. By January 2019, it was clear in industries that hire from elite liberal arts colleges. When we talk to people, they'll say young employees are so fragile and come in looking for confrontation. Members of Gen Z, they are used to an environment in which it's all about displaying what side you're on, supporting this person, opposing that. Everything becomes a moral conflict. But you can't run a company that way.

Q: How do we stop this from happening in an office?

H: If corporate culture becomes political, people feel fearful or reluctant to stand up. We need a speak-up culture, but Gen Z has been raised with a call-out culture in which they're incentivized to criticize or shame people publicly. A callout culture makes people feel they're walking on eggshells, and you can't have that in a company.

So there you have it. This is a cultural problem in tech that needs addressing by multiple organizations. Leaders of the companies should start speaking up against the growing divisiveness in tech, and aim to restore meritocracy.

---

Haidt has summarized a sociological paper to provide a more general sociological perspective into what is happening in tech: Where microaggressions really come from: A sociological account.

Answer 3

it is still a survey instrument

No, it is a diversion, a supposed alternative to actual dialog between the company and the SE community, here on MSE. I suggest Galastel's excellent summary of it.

and reflects upon us all

Cooked surveys which are put forward as a diversion and managed by untrustworthy people/organizations should not be taken seriously.

But I suggest we all make sure and actively undermine this initiative, by boycotting these surveys.

Answer 4

They ask for it because they consider the race or gender of the people who answer is relevant to judge their opinions.

I have a dream that my children will one day live in a nation where they will not be judged by the color of their skin but by the content of their character - MLK

That was back in the 60s, now we are back at racial profiling. And they call themselves progressive inclusive.

Answer 5

I don't know about other sites, but on MathOverflow, there are essentially no women.

Most users use their full names as user names and have photos as avatars and link to their professional web pages so it is clear that they are men. There are essentially no women who identify themselves in this way. There are some pseudonymous users who might be women, but not very many. The proportion of women is far, far below the proportion in the professional mathematical community generally.

There are certainly many women who could greatly enrich the community with their knowledge and insights if they chose to do so. Of course I do not suggest that women should be pressured to participate if they do not find it rewarding. However, if there are fixable incidental reasons why women find the site unwelcoming, then it would be a good thing to understand that.

One could make some guesses about the reasons, but I do not think that any of the most obvious guesses are consistent with the observable data.

For example, in the small number of cases where women do ask or answer questions, I cannot find any sign that their opinions are disrespected or devalued.

So there is a need for further information.

There is some chance that the "The Loop" survey will help with this. It can only do so if it asks about the gender of respondents.

Answer 6

Is it relevant to Q&A? No

Is it relevant to marketing and trying to decide how to advertise? yes

Stack has also taken up an ironic policy of inclusivity so it could be part of an ongoing attempt/operation to make minority groups feel more welcomed by weighing race into the feedback provided.

Doing this lets them market to target groups, however the exclusion of nationality may greatly hamper these capabilities as stated elsewhere, and if it is weighted will let them know which opinions according to their internal workings are more relevant than the others (one of the reasons you have to call it ironic).

My personal belief is that policies should be general and not centered around gender or race, though Age is understandable to the extent of being too low or accessibility for older individuals (should also ask if disabled though if they are considering accessibility).

So in short, It is primarily for marketing and growing the community. I'm a bit new to meta, but it doesn't seem meant to replace it though it also doesn't seem as if they listen to meta as is.

Answer 7

They have reasonable grounds already to suspect that gender and ethnicity will be relevant, based on past annual surveys. As someone else pointed out, and you imply, this looks like an inept attempt to take that into account.

I guess the initial survey asks for only these kinds of demographic information to establish a pool of people they might select from. They might then select a suitable number of people from the various demographic groups.

Answer 8

A potential answer to this occurred to me while reading your question, specifically your section about when demographics should be used:

In scientific research, demographics are only requested if there is evidence that a demographic attribute will lead to a significant difference in responses

Perhaps this is what SO/SE is trying to reveal, is there a difference in user satisfaction based on race/gender/age?

Perhaps 75% of white men will report a positive experience on the site, while non-binary individuals report only 40% positive experiences. (Please note these two groups are not mutually exclusive.)

I am in no way defending the way the survey was done or the lack of appropriate diversity in options, but I do see the above scenario as a possible reason for demographics to be present in the survey.

Answer 9

I presume their objective is to see whether people of different race/age/gender have different experiences of the site.

Answer 10

I can see two reasons why SE wants to collect race, age, and gender.

1. This is useful information if you are trying to sell advertisements
2. One of the objective of SE, is to become more welcoming, in particular to minority groups. This survey can be used to define KPIs regarding this and award senior management with bonuses if their targets are met.

Answer 11

Sadly it is a fact that ethnicity has big influence

Let's be realistic: It is a fact that 'race' (I believe we better call it ethnic background instead) is having influence on social life.

Especially in the United States, where the company resides, one's ancestry (or ethnic background or ethnic identity) has a large impact on one's socioeconomic status.

That is, on a larger population level. An individual person with dark skin can become the president, but the group as a whole is experiencing friction, and this is clearly visible in statistical data.

---

Science/Technology/IT is one field where it plays a role

This influence of ethnic background is protruding into science and technology. If most people in IT are white and male then this might create prejudice in this field.

This occurs on the one hand in stuff like political correct language use in coding which is currently a hot topic. Like Python avoiding the master/slave terminology. But on the other hand there might be also more direct influences of white male 'dominance' in science and technology (like reductions in equal opportunities, salary equality and inclusiveness).

---

It is not inherently offensive for a survey to ask about ethnicity, but it is a sensitive topic and the execution needs to be carefull.

The survey is in principal aligned with a goal of the company to improve inclusiveness and create a more equal level playing field for minorities who may currently experience obstructions.

The goal of the survey seems to be to find out if and how people with a different ethnic identity ('race' is a bit negative term) might be experiencing different problems with the Stack Overflow website (the question four is about "What do you find most frustrating or unappealing about using Stack Overflow?").

Yes, it might be considered a bit harsh to ask about one's ethnicity (especially when the term that is being used is 'race'). It might be considered as 'racist' because it confirms the differences between people with different ethnicity and explicitly treats people with different ethnicity as different.

Yes, there are many ways to criticize the particular execution of the survey and the race/ethnicity concept. The way that SE/SO has categorized 'race' exposes underlying ideas about race and how this interacts with the use of the Q&A platform.

However, it is a fact that people with different ethnic identities are, on a population level, different. This survey has the goal to find out whether there are any negative effects that may follow from that and it will allow to create informed* policy that may diminish those negative effects.

The term 'racism' or 'bigotry' has a negative connotation, but not every differentiation between race/ethnicity or gender (what one might consider in a broader sense racism or bigotry) should be considered negative or intolerant.

---

*It is very easy to create an image of non-inclusiveness based on stories from a small number of people or other anecdotal evidence. The numbers/data can support this image.

---

Diversity

The image below is based on some data from the 2019 Survey. It shows how different the visitors of Stack Exchange and Stack Overflow are. But also, that a majority is 'white male' (from different origin Europe/Russia/USA and this might possibly need to be split up as well) and there are very few women and few people that do not associate with "white" ethnicity.

This makes it problematic to recognize problems that relate specifically to minorities. For this you need to be able to target them specifically, e.g. with a demographics based survey. If you ask the entire group, or listen to the loudest majority, then you may hardly hear the voices of minority groups.

In the image below the correlation with a problem 'belonging to the community' is not so much different for ethnicity (in binary form white vs non-white). But for gender the difference is more clearly different (and reasonably uniform among different countries, thus not influenced by country but rather by 'associating with male gender or not').

And for the aspects that respondents would like to change (which are not available in the open database) there are (many more) clear differences as well. This is sort of what the loop is imaging (based on keywords from open questions). It images how different people think different about SE/SO along with the demographic information. It will give a sharp image , rather than a fuzzy image that mixes/blends everything together. It will be colorful rather gray. It will have depth rather than being flat. (of course, it is arguable what the meaning of that image is and it will only provide additional information and one can not directly read 'causal' relationships from it)

Answer 12

We have all seen SE's recent push to be as welcoming and inclusive of minority groups as humanly possible. The latest controversy involving a certain moderator-who-shall-not-be-named _{^{who happens to have a Go Fund Me set up right now}} started largely because said user asked in good faith for clarification about a proposed rule change involving gender identity.

SE obviously collects data about who uses their site, and they probably already have at least a general idea of the various ethnic backgrounds and ages of developers on their site. That being said, having concrete survey results showing a large percentage of minorities using the site would be quite useful to SE corporate to justify future changes along their current path.

Of course, I say "concrete" survey results, in the sense that they will be numbers. Whether or not those numbers will be skewed as a result of the questioned asked is another matter. I know that I would not bother answering a survey in any context which asked my age, race, or gender, first because I want to hold onto the tiny remnants of privacy we have left. But also because I see them as irrelevant factors on the internet.

However, other people may consider their age, race, or gender INCREDIBLY important, and would take any chance they get to respond to surveys in which they can change the results with their answers. This is a long way of saying that SE most likely included the questions they did to get results closer to what they wanted to see.

Answer 13

They can be used to group the data obtained from the questions "what do you like best" and "what do you find frustrating".

For example, "Lack of fluency in English" can be a common answer for a particular demographic group. Since, the data is already processed through machine learning^ref this can be categorized easily.

Answer 14

Looking at the survey, I really don't understand why demographic information (i.e., race, age, and gender) are being asked since it appears that the clear goal is to get an initial impression of "how satisfied are you with your experience using Stack Overflow?" While I understand that a lot of people will include these by default, unless it is relevant, it's generally taught in survey methods courses to avoid asking the questions.

I'm okay with thinking out loud on why this matters.

Some sample questions to ask of the data:

• Are female developers between the ages of 18-35 feeling like the moderation done on the site is too extreme?
• How satisfied are users younger than 18 with the site compared to users between 18-24?
• Is there a correlation between identified gender and site satisfaction?
• What is the actual demographic make-up of users of the site?
• Is there a correlation between years in the industry and site satisfaction?

How can you ask those questions of your data if you don't actively request it?

Answer 15

I suspect it's because they want to see whether a different subset of users are clicking on the survey compared to their user traffic data, and annual survey data.

It's hamfisted and probably pointless, but at least I can see some motivation behind it.