Unfortunately I need to be a bit blunt since they were poorly designed.1 While it is reasonable to assume that the cultural background of a Stack Overflow (SO) user would result in different perceptions in the use of the website, asking about racial background is a poor way to go about getting that information.
First, there is the obvious problem that asking about "racial background" is a culturally sensitive question internationally. While some of the users of SO are from the United States, the international community is the majority of the user base and needs to be considered. This is fairly easy to correct with better phrasing though.
Second, the bigger issue is that when developing a survey you need to have an idea of what the like dependent and independent variables are.2 In this case the dependent variable (the one being influenced) could be "Perception of Stack Overflow" or "Use of Stack Overflow" and the independent variables (the ones thought to influence the dependent variable) could be drawn from the following list:
- Age: users may be more or less inclined to use SO due to age.
- Professional Experience: more experienced software developer may not use the website as much.
- Cultural Background: the cultural background may play a role; however, this is likely due to a independent variable that relates to cultural background such as native language or English proficiency.
- Gender: are women less like to use SO than men?
- Programming Language: developers who use a underrepresented language on the site may not use it as much.
- Type of Programming: what the developer is programming for may play a role, for example, web developers may be more inclined to use SO versus embedded systems programmers.
The lists of independent variables are usually developed through several sources (ex., prior surveys, literature reviews, brainstorming) and may be narrowed down a bit due to instrument constraints (ex., paper surveys have print costs) or due to overlap on the list. In this case it appears that "The Loop" is using "racial background" as a proxy for the cultural background, but does so in a way that loses a lot of statistical power. For example, "East Asia" is a very big place and includes countries that have a lot of software developers (ex., China, Japan, South Korea). Lumping all of these into one bin presumes there is no significant difference between the countries. This point is a debatable point, and given my familiarity with the region I would argue it's a bad assumption to make.
Given that this was a web-based survey the designer of the survey instrument actually has more tools at their disposal than a paper-based one. As such, I would design the survey instrument to take advantage of this (brainstorming questions):
Q1. In what country do you currently reside? [List of options]
IF [COUNTRY] = [UNITED STATES] THEN Q1.1 What ethnicity do you identify with? [List of options]
Q2. What is your level of proficiency with English? [Likert scale]
This approach captures the same data as the original question in "The Loop" survey since the country selected for Q1 can then be binned into a derived variable for one set of statistical tests. However, by looking at the role that the respondent's country and English proficiency plays it offers a lot more information for analysis. Depending on the results of the analysis a follow-up survey could then be developed that is a bit more targeted.
However, the trick to all of this is that you need to have some hypothesis in mind for the role that the independent variable plays on the dependent variable. During my training, development of causal diagrams played a major role since that in turn acted as the outline for the statistical analysis you would do with the results. It was always interesting to see what independent variables were not statistically significant and it wasn't uncommon to find that you initial hypothesis might be wrong.
- Credentials: PhD with graduate training in computational social science and survey methods.
- Note that for a census your objective is to try and capture a complete representation of user base. The design of a survey instrument changes in that scenario.