24

As a part of an on-going research effort on the impact of collaboration sites on the developers community, we would like to understand the demographics of Stack Overflow participants and their activity. Specifically we are focusing on how genders, minorities and cultural background are represented in the population of users and participants of Stack Overflow.

Therefore, we have prepared a small questionnaire:

https://docs.google.com/spreadsheet/viewform?formkey=dEhtUVNQTEJmRTlwMVJSQ1hkeUZTR3c6MQ

We ask you to indicate your gender, location, educational background and professional experience in Open Source and proprietary software.

Filling this questionnaire should not take more than a couple of minutes. Personal data will not be made available to third parties and no identifiable details about individual participants will be published.

We are:

  • Andrea Capiluppi [Brunel University, UK]
  • Alexander Serebrenik [Eindhoven University of Technology, NL]
  • Bogdan Vasilescu [Eindhoven University of Technology, NL]

Update We would like to thank all Stack Overflow participants for answering our survey. Discussion of the survey data has been included in a paper to appear in the Proceedings of the 2012 ASE International Conference on Social Informatics ([PDF here]). Specifically, the following paragraph is dedicated to the survey.

To obtain insights in the demographics of the SO, we conducted a pilot study. We asked the respondents to indicate the SO userid, gender, age, country of birth, country of residence, highest education level obtained and years of professional experience, as well as involvement in opensource and proprietary software development. We obtained 136 responses, including 123 valid ones (e.g., indicating a SO userid to be uniquely mapped to an individual). Since the responses were obtained voluntarily, composition of the sample is likely to be affected by the selection bias. However, data was obtained to derive quantitative conclusions.

Our first observation is that indeed the lion’s share of the respondents were male: only 11 respondents from 123 have identified themselves as female, and 112 as male. Moreover, we have seen that the respondents have been predominantly involved either exclusively in proprietary software or both in proprietary and open source software (both 46), while the number of exclusively open source developers was lower (16). Remaining respondents either are not involved in software development at all or indicated a more elaborate answer than “yes”/“no”. This means that a priori one could have expected the share of female SO users to be between 1-5% reported for open source projects [8] and 28% of male reported for proprietary software [9]. Finally, we have observed that a significant group of respondents (22 out of 123) no longer resides in the countries of their birth due to personal, professional or educational reasons.

Improve this question
27
  • 15
    You've seen this: blog.stackoverflow.com/2012/02/survey-results right?
    – Flexo - Save the data dump
    Commented Jul 16, 2012 at 6:22
  • 10
    "as part of the on-going research effort", noting the definite article there, what research effort? For whom?
    – Richard
    Commented Jul 16, 2012 at 7:14
  • 7
    "research effort on the impact of social media on the developers community" So, what's got Stack Exchange got to do with it?
    – Time Traveling Bobby
    Commented Jul 16, 2012 at 7:15
  • 25
    "SO is considered as a social media site." ... ... ... ... No, just no.
    – Time Traveling Bobby
    Commented Jul 16, 2012 at 8:08
  • 8
    @UristMcBobby per Wikipedia definition of social media, SO appears to fit doesn't it? "a group of Internet-based applications that build on the ideological and technological foundations of Web 2.0, and that allow the creation and exchange of user-generated content." Wikipedia article even lists SE under Application examples -> Reviews and Opinions -> Community Q&A
    – gnat
    Commented Jul 16, 2012 at 8:35
  • 5
    @gnat: The sentence before that and the first paragraph beg to differ.
    – Time Traveling Bobby
    Commented Jul 16, 2012 at 8:39
  • 5
    Okay, to outline the thought-process: "Social Media", a website to socialize, with focus on the people. "Stack Exchange", a global database of knowledge, with focus on the content which does not care who contributes content, as long as it is good content. The main difference is that at a social media site everyone is welcome...here on Stack Exchange, we boot people out...the hard way if necessary, to ensure quality of the content.
    – Time Traveling Bobby
    Commented Jul 16, 2012 at 8:49
  • 5
    @AlexanderSerebrenik - one of those papers didn't even manage to get the name right.
    – Flexo - Save the data dump
    Commented Jul 16, 2012 at 9:40
  • 5
    @AlexanderSerebrenik: So what? I can also you show you some websites which state the Facebook is initiated by the CIA to spy on people...just because other people got it wrong, too, doesn't make it right. We, the people of Stack Exchange, tell you: We're not a social site/media in any way...it's neither the goal, nor the foundation of Stack Exchange.
    – Time Traveling Bobby
    Commented Jul 16, 2012 at 9:54
  • 3
    @Richard: Alexander and I have been collaborating in (trying to) produce some value in our research. We tried to focus on produce an h-index for open source developers (scholar.google.com/…), and we have a paper under review about how web activity (also StackOverflow's) could be used by candidates as their resume's. The purpose of this present research is something else, and related to gender, but again we hope to create some value to it, hopefully not waste people's time...
    – user190780
    Commented Jul 16, 2012 at 10:38
  • 2
    @AndreaCapiluppi Could you edit the Q to include those details. (And to read my earlier comment another way: why should I give you some – however little – of my time?)
    – Richard
    Commented Jul 16, 2012 at 10:45
  • 3
    @UristMcBobby: point taken. There's something about SO as a soc... collaboration site that is quite peculiar, in terms of its participants, specifically when it comes to badges, prizes, reputation and so on. I'd love to hear what your experience has been.
    – user190780
    Commented Jul 16, 2012 at 10:58
  • 5
    @AndreaCapiluppi: Badges and reputation are not social, they're egoistic. They're not for the people, they're for me. If you want to learn about the community, spend half a year on Stack Overflow, Meta and a new SE2.0 site, like Bicycles.
    – Time Traveling Bobby
    Commented Jul 16, 2012 at 11:09
  • 3
    @UristMcBobby: that's exactly what we're trying to prove (and please assume that we do have some experience in online communities)
    – user190780
    Commented Jul 16, 2012 at 12:03
  • 2
    @ShegitBrahm - because we want to match your gender/education level/country with your activity at SO as measured by reputation, number of questions/answers/comments, presence/absence/number of certain badges.
    – Alexander Serebrenik
    Commented Jul 19, 2012 at 9:37

2 Answers 2

Reset to default
48
+50

Thank you for taking an active interest in the Stack Overflow/Stack Exchange community.

As you've seen from the comments, the members of the Stack Exchange community can be passionate about how they're defined by the outside world, hence some of the negative downvoting and comments.

On behalf of the community, I apologize for that. As an academic who is seeking knowledge and trying to contribute to the growing knowledge base of how the world works, you share some of the goals of the Stack Exchange sites.

I haven't seen the survey, but others have pointed out that they aren't comfortable with the characterization of the site in the same light as other sites (such as Facebook, Twitter, etc).

The members of Stack Exchange have typically eschewed such analogies, even though the gaming elements are more than likely the main cause to spur people to produce good content.

Personally, I don't have a problem with the characterization that you've portrayed, but if you are looking for more active participation in your study from the Stack Exchange members, you might want to take the constructive elements of the comments into account for the framing of your survey.

To the commenters: Be nice.

These are people who have goals that are not nefarious by any means and are taking interest in Stack Exchange's community. Let's not forget a huge part of our core values, especially to outsiders, just because they have a different opinion than our own on what exactly it is that we are.

To this point, I've seen Alexander et al. be nicer to us than we have to them. Let's put a better foot forward here.

Improve this answer
12
  • 1
    Thank you very much for your answer. We have modified the motivation part of the questionnaire to reflect the positive suggestions made by the commenters. Specifically, we no longer talk about social media but about collaboration sites. We are always glad to learn from the Stack Overflow community.
    – Alexander Serebrenik
    Commented Jul 16, 2012 at 13:13
  • 6
    @AlexanderSerebrenik If you could put a little bit of the questionnaire (specifically, any goals/characterizations/assumptions/what you're trying to accompish) from the link into the question up top, it would help tremendously, and I think you'll find the collaborative nature of the SE community will provide even more positive feedback in order to elicit more participants in your study.
    – casperOne Mod
    Commented Jul 16, 2012 at 13:15
  • 3
    Thank you very much for your suggestion. I've modified the title to make it more specific and added an indication of what kind of questions we ask. Unfortunately, I cannot explicitly state the hypothesis we are trying to confirm/reject as it might motivate the participants to give the answers the researcher would like to hear, as opposed to the true ones ;-)
    – Alexander Serebrenik
    Commented Jul 16, 2012 at 13:22
  • 3
    @AlexanderSerebrenik Fair enough, thank you for trying to work with us despite some brusque behavior. It's much appreciated.
    – casperOne Mod
    Commented Jul 16, 2012 at 13:24
  • 1
    @AlexanderSerebrenik - out of interest are you able to commit to promise to publish the findings (and data) in an open access repository?
    – Flexo - Save the data dump
    Commented Jul 17, 2012 at 9:49
  • 4
    @Flexo: we can publish the core findings, for instance, on meta (shall I then edit this discussion or shall I start a new "question"?). Data should be anonymised and presented very carefully due to privacy concerns: certain combinations of gender/age/country might completely identify the respondent.
    – Alexander Serebrenik
    Commented Jul 17, 2012 at 11:20
  • @AlexanderSerebrenik I would publish the results here, as an addendum to the original question (do not edit out the old question, except maybe near the link to indicate that the study is complete).
    – casperOne Mod
    Commented Jul 19, 2012 at 20:42
  • @casperOne Thank you!
    – Alexander Serebrenik
    Commented Jul 19, 2012 at 20:52
  • @casperOne From the first time I saw this question, when she was still fresh new, till now!!... I can't stay without manifesting my contentment! Well done! Well done! :D +1
    – Zuul
    Commented Aug 1, 2012 at 6:20
  • @casperOne I have added a quote from the paper we have recently submitted discussing our findings related to the survey.
    – Alexander Serebrenik
    Commented Aug 23, 2012 at 3:29
  • Is negative downvoting the same as upvoting?
    – gerrit
    Commented Feb 19, 2013 at 16:38
  • @gerrit I don't know. On meta, voting is just too convoluted, as there's a duality to the votes.
    – casperOne Mod
    Commented Feb 19, 2013 at 18:15
19
+50

Selection Bias in Action

There's going to be a self-selection bias in your survey results. I'm sure you already know that, so I won't belabor it.

The reason I'm breaking out my response as an answer, not a comment, is so that I can address why I am self-selecting out of the survey. My reasons are:

  1. It's not anonymous. I understand the trade-offs involved in anonymous surveys, but I don't like to participate in surveys that gather more information about me than I choose to provide in the survey itself.

  2. I consider some of the questions (specifically age, location, and birth record data) to be personal, especially since they survey itself is not anonymous. Not everyone may feel that way, but I do.

  3. The linkage to the user account enables data mining that is not explicitly laid out in the questionnaire. Whether or not this is intended, it's certainly possible, and that makes me personally uncomfortable. Even though Stack Exchange user accounts make most information public, you have to know how to look for the information; that provides for a certain amount of "star in the sky" anonymity that account linkage takes away.

Summary

I'm not criticizing the survey design. It's your survey, and the results may be useful. I'm simply offering some thoughts on possible self-selection considerations, and highlighting the fact that "living in public" in a Web 2.0 world isn't quite the same thing as not valuing one's privacy or personal data.

Your mileage, and the mileage of any self-selected survey takers, may certainly vary. With that in mind, I wish you success with your survey!

Improve this answer
4
  • 3
    Thank you for your response. 1) Since the survey is only one of the techniques we intend to use, and it will be augmented by analysing the data publicly available in the SE dump, I believe the selection bias is appropriately addressed. 2) We do value your privacy and personal data, and, hence, as explained above we will not publish any personal information. 3) Should participants prefer not to answer some of the questions, they also can choose to provide a "Prefer not to answer" as an answer.
    – Alexander Serebrenik
    Commented Jul 19, 2012 at 19:50
  • 4
    @Alexander you are not offering "prefer not to answer" as an option in the text fields though.
    – Pekka
    Commented Jul 21, 2012 at 7:34
  • 6
    @Pekka: this is, unfortunately, a limitation of GoogleDocs. However, I've added "if you prefer not to answer, please enter ..." as an alternative.
    – Alexander Serebrenik
    Commented Jul 21, 2012 at 7:40
  • @Alexander cool, I think that makes it clearer.
    – Pekka
    Commented Jul 21, 2012 at 7:48

Not the answer you're looking for? Browse other questions tagged .