46

It seems non-coincidental that the paper by May, Wachs, and Hannák (2019) was published online on the 8-th February 2019, shortly prior to the recent change to question reputation points (13-th November 2019). The authors proposed and analyzed the changes that were subsequently implemented:

... we discuss one potential way to mitigate the gender differences in outcome and success on Stack Overflow: modifying the reward system. Our results suggest that differences in the rate at which men and women post answers account for the largest part of the reputation gap. On the other hand, women tend to ask more questions than men. ...

Increasing rewards to good questions may help to make the site more inclusive by offering a less competitive and speed-oriented way to build one’s reputation. ...

To calculate the revised reputation score, we collect additional data on each user’s activity. We calculate the revised reputation scores of all users by counting question upvotes as being worth 10 points.
Section 6 of May, Wachs, and Hannák, Gender differences in participation and reward on Stack Overflow, 2019.

I don't believe the relationship has thus far been explained.

Question: What is the relationship between the paper by May, Wachs, and Hannák (2019) and the recent changes to question reputation?

  • 3
    That's just over 9 months from publishing to the change in question reputation. As that doesn't fall within a 6-8 weeks or 6-8 months range, it can't possibly be related, right? =) – Rob Nov 15 at 10:09
  • 32
    If I understand the "6 to 8 weeks" meme correctly, 9 months falls safely within a "6 to 8 weeks" timeframe. – Rebecca J. Stones Nov 15 at 10:11
  • 1
    I guess the "less... speed oriented" is a reference to the Fastest Gun in the West Problem. – Raedwald Nov 15 at 10:12
  • 30
    I read through the paper and there's a good reason it wasn't included - it had a few rather gaping holes in methodology that would probably have been big enough to drive a truck through. It basically had a monumental bias I will leave to the reader to work out. – Journeyman Geek Nov 15 at 10:12
  • 1
    It seems that this has been mentioned in internal discussions (and discussions with moderators) and this link has been posted in one of previous questions here. However this may all be just hearsay. On the other hand I would not be surprised if this kind of thing also had an influence on decision. Because SE is all about PC now, regardless on detrimental effects their PC-ing has on particular groups. – Resistance Is Futile Nov 15 at 10:13
  • 2
    The paper also suggests that further drastic measures are necessary to ameloriate the gender gap, so it's interesting to read in the sense of what might be coming next, especially now that they have a robust roadmap they're not taking feedback about. – Magisch Nov 15 at 10:19
  • 2
    @mag Technically, it cannot be even called gender gap, because it would imply that women are discriminated (rewarded different reputation) for equal contributions. Which is clearly not the case here. Not to mention that drawing any kind of statistics from SO is exercise in futility. There is so much discrepancy between reputations that are gained depending on tags in which person participates and for how long. – Resistance Is Futile Nov 15 at 10:35
  • 1
    Whatever methodological and statistic flaws that paper might have (even I can spot one of them), I guess its findings are better justified than the original decision to reduce the reputation awarded for questions, which the recent change reverses. – Raedwald Nov 15 at 10:36
  • 7
    @Raedwald what you say makes no sense. If the science is not sound, then you can't draw any conclusion to whether it's "better justified" or not, since the justification would be invalid. It might have more justification as in amount but if the quality is poor, it's no different than any other justification. – VLAZ Nov 15 at 10:40
  • 2
    @JourneymanGeek What's the flaw you're referring to? The one that strikes me is that they don't consider users' age or seniority: we know from surveys that older or more experienced devs on SO post more answers, and younger or less experienced devs post more questions; and that the age distribution for female SO users is on average younger (thanks in part to successful initiatives encouraging more girls and young women to pursue STEM careers and coding). Even that study finds that most of the apparent gender difference can be explained by differences in "account age" (days since registering). – user56reinstatemonica8 Nov 15 at 11:09
  • 16
    They use usernames to figure out gender. That's sort of literally means folks with female or male names off a list? – Journeyman Geek Nov 15 at 11:12
  • 4
    Here's a chart of years' experience by gender from the survey the study references, showing that female SO users include a higher % early in their careers (good news we'd expect if initiatives in schools, colleges etc are working). In fact, this difference looks larger than the 11% "unexplained gender gap" the study found - suggesting users with apparently-female usernames post more answers and fewer questions than apparently-male users with the same level of experience? – user56reinstatemonica8 Nov 15 at 11:30
  • 2
    Without meaning to be rude, I don't understand this. Isn't there a spectrum of genders? Why focus on M/F? – Justice for Monica Nov 15 at 13:14
  • 3
    @user56reinstatemonica8 Alternate interpretation of the paper: "Long-time female users of Stack Overflow tend to use ambiguous usernames". – Disenchanted Lurker Nov 15 at 13:27
  • 1
    @JourneymanGeek Judging by upvotes on your comment, the community would love to see your analysis of the paper. It doesn't exactly answer the question, but I think there's no better place to discuss it. – Athari says Reinstate Monica Nov 15 at 14:17
32

The closest so far to an official comment comes from Robert Cartaino, and is no: apparently the change was planned before the staff involved became aware of that study; that study was mentioned as a bonus benefit of the change, then quickly un-mentioned following moderators pointing out its many flaws:

  • It's interesting that they didn't mention anything about doing this to benefit women, since women ask more questions than men, as was the reason given in the leak. Did the mod feedback cause the removal of that reason? – House- 'Reinstate Monica' -man

  • @House Yes, definitely. The genesis for this change goes WAY back to discussions about restoring parity between questions and answers. I didn't hear about this study until much later, so yes, it was best not to jeopardize the launch by trying to anticipate that benefit -- See stackoverflow.com/c/moderators/questions/1912/…Robert Cartaino ♦

I can't follow that link (moderators only?) and haven't seen the discussion or the leak, so I can't comment on whether this comment is plausible or consistent with what was said there.

Jon Ericson has also posted on the English meta claiming that he and others have wanted this change "for years", which would predate this study - and that the reason was that they felt halving rep for questions had proved a mistake:

This change has been in the works since August (according to the date of the initial spec) and we'd been considering it for several months before that. Personally I've been hoping for this change for years. (And I'm not the only one.)

...

The main reason for the change was to correct what we believe was a mistake to change the reputation payout in the first place. That it also makes people feel better is something of a pleasant side-effect.

However, many users who saw that staff/moderator discussion talk as if this study was cited as a (or, the) motivating factor.

To me, it seems all these comments are consistent with the following theory:

  • Many experienced members of SE staff, who are also long-standing users of the SE network, felt that halving rep for questions had not achieved its stated aims in somehow improving question quality, and should be reversed to encourage and facilitate involvement from users who write good-quality questions
  • Senior decision-makers at director level, who seldom use the SE network and have much less experience with it, were not interested in prioritising this change...
  • ...until they saw this study. After seeing this study, they gave this proposal the green light. This would fit Jon's timeline where it was under active consideration "several months" before August (i.e. from around March/ April?

If this is true, the study would be what got the proposal approved and signed off, but wasn't what motivated the proposal in the first place.

It would just be part of how they presented it to the boss in order to get it signed off (we've all been there!).

And it was part of how they presented it to moderators too, until they remembered that moderators do things like actually read studies, and notice if those studies presume gender based on usernames, exclude all users without gendered-real-name usernames, and presume that if a gender group that includes a higher % of early career developers asks more questions than a group that is on average older, it must be because of their gender somehow, not because people early in their careers inevitably have more questions.


If anyone has more insider knowledge or can view the moderator discussion and sees something that contradicts this theory, please comment.

14

When reading through the paper a few specific parts of it felt problematic for me

More specifically, we find significant gender gaps in activity: women are more likely to ask questions, while men provide more answers and cast more votes.

Which I guess could be an observation, but I didn't really find a great reason for the why and how we can help women actually fill that gap. Cause I'm pretty sure gender isn't a barrier to knowledge.

Using the Oaxaca-Blinder decomposition (Oaxaca 1973), a method from economics that to the best of our knowledge has not yet been previously applied to measure gender disparities in online communities, we decompose the outcome differences between men and women in terms of differences in their activity.

This bit is interesting. Unfortunately I'm not smart enough to actually get what Oaxaca-Blinder decomposition is, though since its novel to this application - I'm a little skeptical.

In the final part of the paper, we explore the consequences of a hypothetical redesign of the site’s reward system. The proposed alternative scoring system equalizes the rewards for well-liked questions and answers, a simple and justifiable change which does not penalize any group of users in absolute terms.

This sounds shiny.

Stack Overflow is itself a well-studied platform. Vasilescu et al. show that women are underrepresented in this community (Vasilescu et al. 2013). Interviews with a sample of Stack Overflow users highlight the barriers women have to greater participation. Women respondents listed the lack of awareness of some site features, the intimidating community size and their fear of lacking adequate qualifications as main barriers to participation (Ford et al. 2016).

Now this literally is a problem worth solving IMO. Why are these barriers there? How do we mitigate them? Getting engaged users is essential if you want a constant, vibrant representation of a specific segment of the community

Now the big problem I have with this paper

"We found that while the method performed very well on men (97% agreement with our manual check), our manual check agreed only in 44 out of 100 cases of women. This replicates the recent finding by Ford et al. (2017) that genderComputer sacrifices precision for greater recall when inferring women users."

They did run further rounds - but that also means that feels like selecting for women who use obviously female names, and men who use obviously male names. This basically doesn't select for anonymous users - and someone uncomfortable with being identified by their gender would do that or use an ambiguous name

Here's the frustrating part. I can't share more. Unlike our rogue moderator - I do respect the sanctity of folks trusting you not to share things.

Its worth considering, this wasn't in the final announcement. At best it was one of many things considered both in internal messaging and for external messaging.

  • 2
    I also have to wonder about their identification methodology as it is a known fact that female users use male pseudonyms to avoid conflict so it can actually be classifying women as men. – NathanOliver- Reinstate Monica Nov 15 at 14:56
  • 6
    @NathanOliver-ReinstateMonica Also, men have been known to create attractive female profiles to get better responses in male-dominated online spaces. – ColleenV parted ways Nov 15 at 15:35
  • 1
    SE have been quick to blame an unknown moderator for the leak; but have not, as far as I know, presented or even alluded to any evidence for that allegation. Unless as a moderator on Meta you're privy to more information than the rest of us, it seems presumptious to talk of a "rogue moderator". – Scortchi - Reinstate Monica Nov 15 at 17:18
  • 5
    @Scortchi-ReinstateMonica The leak was a screenshot of a post that only staff and moderators can access. Either it was a rogue staff member who wanted to make their own job harder and likely risk getting fired... or it was a moderator. – Em C Nov 15 at 18:04
  • 1
    @EmC: It doesn't seem entirely implausible that an SE employee might have had the same axe to grind as the moderator's supposed to have had; & be as likely, or more likely, to know how to take elementary precautions against getting caught. Or have had a different motive: to discredit moderators. Better not to jump to conclusions. – Scortchi - Reinstate Monica Nov 16 at 1:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .