51

This question is a follow up to a longer comment thread, the gist of which is that the approach that Stack Overflow uses to develop surveys can introduce significant bias into the results. Social scientists are trained to avoid this bias as well as being trained to explore why communities react the way that they do. While I know that Stack Overflow employees a number of data scientists as well as some people in marketing that have some experience with surveys, neither of these is quite the same having a dedicated social scientist on staff.1

There has been increasing research out of the scholarly community as well that a transdisciplinary approach (e.g. software developers, UI/UX experts, and social scientists) are needed to avoid bias in software development and "black box" decision making algorithms. Given Stack Overflows prominence on the internet it seems like they would be remiss to not employ a fairly diverse team in terms of their professional training. As such, does Stack Overflow employ any social scientists?

  1. I recall a conference where a toxicologist underscored the need to have a social scientist as part of the research team since they got at the crux of why people were persisting in using environmental contaminants.
Improve this question
2

1 Answer 1

Reset to default
36

Yes, we do employ several social scientists. They participate in the planning and execution of projects whenever is appropriate. There are many factors that go into their work, and while they have loads of experience in this domain, I do not, and am definitely not qualified to converse on the subject.

I followed up with my colleagues, and am paraphrasing their responses

  • Our social scientists have years of training in this very kind of social science background, to the PhD level and beyond, and have lots of hands-on experience (academic, industry, consulting, etc) with survey methods and analysis in market research
  • Dealing properly with a global audience in surveys is a hard problem, and one we have been working on for years
  • A good deal of thought has gone into the race question (the version that is up there right now). What we have now is similar to the way that it appears on the annual dev survey (which itself was reframed to work more with international users following user research)
  • There are legitimate concerns with this particular survey's bias as it's not a random sampling but rather feedback from people deciding to opt in. That will be taken into account when we think about how we use the data
  • It is hard to accept statements implying that there is definitely a 100% correct way to run a given type of research. Literally every field disagrees on what constitutes “good” survey questions and “good” qualitative survey methods. There’s no one method to rule them all
Improve this answer
18
  • 31
    Were any of them involved in designing the loop survey?
    – ColleenV
    Nov 26, 2019 at 15:29
  • 4
    Re: self-selection in The Loop survey - yes, we are aware of it. This will be accounted for during analysis of the data. Something that we sacrifice in order to get wider exposure and more responses. We do also have the (very similar) ongoing Site Satisfaction survey which is shown to a small random percentage of anonymous and logged-in SO users every month, which might balance this out a bit.
    – Yaakov Ellis StaffMod
    Nov 26, 2019 at 17:59
  • 20
    I guess we'll never know about the tone of those other comments, since they are all gone now. Does anyone worry about anything else besides "tone" anymore? There's nothing more aggravating than "I think your ideas might have some merit, but I'm dismissing them because I don't like the way you presented them."
    – user102937
    Nov 26, 2019 at 20:24
  • 10
    12 downvotes on an answer to a purely fact-based question. Meta is fundamentally wrong.
    – This_is_NOT_a_forum
    Nov 27, 2019 at 1:48
  • 11
    @PeterMortensen - Read the first version. My vote was a response to the unnecessary hostility in the second paragraoh.
    – anongoodnurse
    Nov 27, 2019 at 2:54
  • 7
    I have posted an update. Based on the answers I received and what I posted here, they are not going to get into a discussion over justifying or explaining “why” one specific approach was chosen over another. Thanks for your understanding. I hope that this has been helpful.
    – Yaakov Ellis StaffMod
    Nov 27, 2019 at 21:03
  • 5
    @PeterMortensen If the answer had been the single word "Yes", then it would be equally valid ... and similarly vague: It does not say much more than "It's hard, there is no right way of doing it, but we have experts, trust us". It just reads too much like an attempt to deflect any concerns that could be brought up regarding the 1. purpose of a survey, 2. the methods for analyzing the results, and 3. the conclusions that are drawn, and 4. the policies that are implemented based on that. This answer does not help SE to regain trust (although it's likely not supposed to do so).
    – Marco13
    Nov 27, 2019 at 21:18
  • 6
    "That will be taken into account when we think about how we use the data." What do you mean by "use" here? Do you mean "analyse"? Because you really should think about that before you collect the data. (I understand you are just the messenger, but this really sets off alarm bells.)
    – user630245
    Nov 27, 2019 at 21:37
  • 12
    Thank you for engaging with the community and updating your response. I know it can be challenging, but this is exactly the kind of approach needed to repair the relationship between the SE and the community. Even though this answer may continue to get votes and comments expressing disagreement, I hope you can see how much your engagement is appreciated.
    – De Novo
    Nov 27, 2019 at 22:13
  • 3
    @YaakovEllis the final revision of your answer is very much appreciated!
    – Anton Menshov
    Nov 28, 2019 at 0:34
  • 3
    Could you add an updated screenshot of this updated question (currently those of us who already took it cannot see what the update looks like). Also, to reinterate my question for the third time to the third person (because it seems any questioning of the blog post results in your comment being blocked) - what lead to the white supremacist language of connecting European and White used in this survey? (I honestly hope someone just says: "we were trying to ask nationality and race & shouldn't have combined them" which is still bad but at least not intentionally bad)
    – LinkBerest
    Nov 28, 2019 at 3:24
  • 13
    "when we think about how we use the data" - which as every good scientist knows, must happen before you collect the data
    – OrangeDog
    Nov 28, 2019 at 16:57
  • 6
    You are mixing racial and geographical terms on the racial question of the survey. White is not a race hispanic is not a race. East Asia is not a race. You should ask for country or continent instead
    – user657339
    Nov 30, 2019 at 21:33
  • 10
    The race question was not well thought through in my opinion. The initial version was incorrect, and it then got updated several times within a few hours of posting the questionnaire. That looks more like crisis management, rather than a thorough scientific evaluation. I appreciate the data scientist may not have been given sufficient time. In that case why didn't they cope ethnicity categories from an USA government body?
    – Alex
    Nov 30, 2019 at 21:47
  • 10
    @Marco13 "we have experts, trust us" is difficult to accept when said experts forget black people exist, let alone all the other issues with the loop survey. I'm not even sure experts were involved in that at all, which shows the board being even less competent than even we though.
    – gbjbaanb
    Nov 30, 2019 at 23:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .