How were the racial backgrounds used by The Loop and previous surveys designed?

Question

In the current version of "The Loop", the following question is asked:

5. Which racial background(s) do you identify with? Please select all that apply. (optional)

Multiracial
Hispanic or Latino/Latina
Biracial
Native American, Pacific Islander, or Indigenous Australian
Middle Eastern
South Asian
Black or of African decent [typo in original]
White or of European descent
East Asian
Other (please specify)

Similar questions have been used in past surveys, for example this 2017 survey mentioned the following, referred to as ethnicities in the writeup as opposed to racial backgrounds:

• Hispanic or Latino/Latina
• Native American, Pacific Islander, or Indigenous Australian
• Middle Eastern
• South Asian
• Black or of African descent
• White or of European descent
• East Asian
• I prefer not to say
• I don’t know

The groups mentioned above are somewhat different to those used by the US census, the country in which the company is based. It mentions the following (copied from the list of 2000, which apparently hasn't changed for 2010):

• White
• Black or African American
• American Indian and Alaska Native
• Asian
• Native Hawaiian and Other Pacific Islander
• Some other race
• Two or more races

It also asks about the ethnicity "Hispanic or Latino".

There's a number of differences between what the company does, and what the US census bureau does. The survey groups "American Indian and Alaskan Native" with "Native Hawaiian and Other Pacific Islander". It splits the "Asian" category up. It treats "Hispanic or Latino/Latina" as racial background rather than a separate attribute. Most interestingly for me, it groups members of "Indigenous Australian" with "Native American" and "Pacific Islander".

How did the people behind "The Loop", and previous surveys, decide upon the categorisation of racial backgrounds?

Update In the 2020 developer survey, the following option was given, to a question which didn't use the word "racial" or "ethnicity":

Indigenous (such as Native American, Pacific Islander, Or Indigenous Australian)

This would suggest that in 2019, the designers of The Loop thought back then that the three groups had that characteristic in common.

Answer 1

Unfortunately I need to be a bit blunt since they were poorly designed.¹ While it is reasonable to assume that the cultural background of a Stack Overflow (SO) user would result in different perceptions in the use of the website, asking about racial background is a poor way to go about getting that information.

First, there is the obvious problem that asking about "racial background" is a culturally sensitive question internationally. While some of the users of SO are from the United States, the international community is the majority of the user base and needs to be considered. This is fairly easy to correct with better phrasing though.

Second, the bigger issue is that when developing a survey you need to have an idea of what the like dependent and independent variables are.² In this case the dependent variable (the one being influenced) could be "Perception of Stack Overflow" or "Use of Stack Overflow" and the independent variables (the ones thought to influence the dependent variable) could be drawn from the following list:

1. Age: users may be more or less inclined to use SO due to age.
2. Professional Experience: more experienced software developer may not use the website as much.
3. Cultural Background: the cultural background may play a role; however, this is likely due to a independent variable that relates to cultural background such as native language or English proficiency.
4. Gender: are women less like to use SO than men?
5. Programming Language: developers who use a underrepresented language on the site may not use it as much.
6. Type of Programming: what the developer is programming for may play a role, for example, web developers may be more inclined to use SO versus embedded systems programmers.

The lists of independent variables are usually developed through several sources (ex., prior surveys, literature reviews, brainstorming) and may be narrowed down a bit due to instrument constraints (ex., paper surveys have print costs) or due to overlap on the list. In this case it appears that "The Loop" is using "racial background" as a proxy for the cultural background, but does so in a way that loses a lot of statistical power. For example, "East Asia" is a very big place and includes countries that have a lot of software developers (ex., China, Japan, South Korea). Lumping all of these into one bin presumes there is no significant difference between the countries. This point is a debatable point, and given my familiarity with the region I would argue it's a bad assumption to make.

Given that this was a web-based survey the designer of the survey instrument actually has more tools at their disposal than a paper-based one. As such, I would design the survey instrument to take advantage of this (brainstorming questions):

Q1. In what country do you currently reside? [List of options]

IF [COUNTRY] = [UNITED STATES] THEN Q1.1 What ethnicity do you identify with? [List of options]

Q2. What is your level of proficiency with English? [Likert scale]

This approach captures the same data as the original question in "The Loop" survey since the country selected for Q1 can then be binned into a derived variable for one set of statistical tests. However, by looking at the role that the respondent's country and English proficiency plays it offers a lot more information for analysis. Depending on the results of the analysis a follow-up survey could then be developed that is a bit more targeted.

However, the trick to all of this is that you need to have some hypothesis in mind for the role that the independent variable plays on the dependent variable. During my training, development of causal diagrams played a major role since that in turn acted as the outline for the statistical analysis you would do with the results. It was always interesting to see what independent variables were not statistically significant and it wasn't uncommon to find that you initial hypothesis might be wrong.

---

1. Credentials: PhD with graduate training in computational social science and survey methods.
2. Note that for a census your objective is to try and capture a complete representation of user base. The design of a survey instrument changes in that scenario.

Answer 2

I think that part of the survey was implemented following Hanlon's razor:

"Never attribute to malice that which can be adequately explained by stupidity."

But let's better call it incompetence, as we are professionals here.

As in:

The groups mentioned above are somewhat different to those used by the US census

To me, the only place in the world where my "race" matters is "US customs and immigration" when traveling.

And that is the thing: asking such a question isn't just "sensitive internationally". No kidding: if SE Inc. would be a Germany company, and they had pulled that survey, they would have made it the national stage, and they would have seen a 2 week shit storm. Minimum. They wouldn't have gotten away without the CEO publicly apologizing for being so insensitive, unwelcoming and racist.

Given the fact that SE Inc. over and over emphasises a global community, and put major efforts in being "welcoming", that part of the survey simply indicates one thing: cluelessness.

Regarding "how it happened", what we know is:

• SE Inc. postponed the initial announcement (and going public with their survey) by a few days, to fix "problems". Only God knows bad the original content must have looked like.
• They really don't care for feedback from this community. If those US Americans pulling together that survey had talked to a say, a few European users ... I am sure: that question about race wouldn't have been in the final survey. At least not like this.

But I fear, that is the way how things work now: SE Inc. has highly paid professionals defining the new path forward. And because they are soooo good at it, they can't get anything wrong, therefore there is no need to listen to us upfront.

Answer 3

The groups mentioned above are somewhat different to those used by the US census

Of course. Because such attempts to group are always built on pseudo-biological theory.

Attempting to categorise individual humans by race is a highly dubious pseudo-science which first sought acceptance in the 16th century's Age of Exploration (for reasons we can readily guess) and which led to some truly horrific theories and practices in the 17th, 18th, 19th and 20th centuries.

As a global population we would all do well to regard the proposal that homo sapiens is divided into races with the withering scepticism such a flawed notion merits.

Answer 4

How did the people behind "The Loop", and previous surveys, decide upon the categorisation of racial backgrounds?

...

The groups mentioned above are somewhat different to those used by the US census, the country in which the company is based.

The blog: Introducing “The Loop”: A Foundation in Listening points to: "Developer Survey Results 2019" which points to: "State of the Stack 2019: A Year in Review", which says they use the Rooney Rule (as does Pinterest, Facebook, Patreon and Checkr, etc.,) during hiring.

"We’ve been working hard to create a more diverse team and inclusive team internally. We’re taking a look at our policies, benefits, interview practices, and internal trainings to make sure that we’re supporting the team we have. We’re also working hard to recruit a more diverse team.

...

While this isn’t where we want to be, it represents substantial growth over the past few years. In 2018, we also concluded a third-party compensation review to address any gender or racial pay gaps, and implemented a Rooney Rule for tech and leadership hiring to ensure we continue to address the gaps in our hiring.".

Whether hiring practices extend to every possible interaction isn't specified but fair application of the rule would ensure that population demographic groups and Stack Exchange/Overflow demographic groups are held at approximately equal levels. Were there any shortcoming it ought to be investigated. With the Rooney Rule in effect and that blog article co-authored by two persons, subject to review (internal and external), one expects that efforts are being made to equalize representation and treatment of everyone.

That isn't without its difficulties. The manner in which it is split needs to be categorically factorable. Categorization, even within the US Census, has been a fluid and open question, which is often reanalyzed.

On page 4 of "American Anthropological Association Response to OMB Directive 15: Race and Ethnic Standards for Federal Statistics and Administrative Reporting" (Sept 1997) an analysis of how categories are derived for the US Census was presented:

"The American Anthropological Association recognizes that classical racial terms may be useful for many people who prefer to use proudly such terms about themselves. The Association wishes to stress that if biological information is not the objective, biological-sounding terms add nothing to the precision, rigor, or factual basis of information being collected to characterize the identities of the American population. In that sense, phasing out the term "race," to be replaced with more correct terms related to ethnicity, such as "ethnic origins," would be less prone to misunderstanding.

Social and Cultural Aspects of "Race" and "Ethnicity"

Race and ethnicity both represent social or cultural constructs for categorizing people based on perceived differences in biology (physical appearance) and behavior. Although popular connotations of race tend to be associated with biology and those of ethnicity with culture, the two concepts are not clearly distinct from one another.

While diverse definitions exist, ethnicity may be defined as the identification with population groups characterized by common ancestry, language and custom. Because of common origins and intermarriage, ethnic groups often share physical characteristics which also then become a part of their identification--by themselves and/or by others. However, populations with similar physical appearance may have different ethnic identities, and populations with different physical appearances may have a common ethnic identity.

OMB Directive 15 views race and ethnicity as distinct phenomena and appropriate ways to categorize people because both are thought to identify distinct populations. Although this viewpoint may capture some aspects of the way most people think about race and ethnicity, it overlooks or distorts other critical aspects of the same process.".

Let's see what happens when we take organizations that we presume to be fair and honest and ask about their demographics, does this look even?

^{Source: "Police Department Race and Ethnicity Demographic Data" (Click to Zoom)}