What is the relationship between the paper by May, Wachs, and Hannák (2019) and the recent changes to question reputation?

Question

It seems non-coincidental that the paper by May, Wachs, and Hannák (2019) was published online on the 8-th February 2019, shortly prior to the recent change to question reputation points (13-th November 2019). The authors proposed and analyzed the changes that were subsequently implemented:

... we discuss one potential way to mitigate the gender differences in outcome and success on Stack Overflow: modifying the reward system. Our results suggest that differences in the rate at which men and women post answers account for the largest part of the reputation gap. On the other hand, women tend to ask more questions than men. ...

Increasing rewards to good questions may help to make the site more inclusive by offering a less competitive and speed-oriented way to build one’s reputation. ...

To calculate the revised reputation score, we collect additional data on each user’s activity. We calculate the revised reputation scores of all users by counting question upvotes as being worth 10 points.
_{Section 6 of May, Wachs, and Hannák, Gender differences in participation and reward on Stack Overflow, 2019.}

I don't believe the relationship has thus far been explained.

Question: What is the relationship between the paper by May, Wachs, and Hannák (2019) and the recent changes to question reputation?

Answer 1

The closest so far to an official comment comes from Robert Cartaino, and is no: apparently the change was planned before the staff involved became aware of that study; that study was mentioned as a bonus benefit of the change, then quickly un-mentioned following moderators pointing out its many flaws:

• It's interesting that they didn't mention anything about doing this to benefit women, since women ask more questions than men, as was the reason given in the leak. Did the mod feedback cause the removal of that reason? – House- 'Reinstate Monica' -man

• @House Yes, definitely. The genesis for this change goes WAY back to discussions about restoring parity between questions and answers. I didn't hear about this study until much later, so yes, it was best not to jeopardize the launch by trying to anticipate that benefit -- See stackoverflow.com/c/moderators/questions/1912/… – Robert Cartaino ♦

I can't follow that link (moderators only?) and haven't seen the discussion or the leak, so I can't comment on whether this comment is plausible or consistent with what was said there.

Jon Ericson has also posted on the English meta claiming that he and others have wanted this change "for years", which would predate this study - and that the reason was that they felt halving rep for questions had proved a mistake:

This change has been in the works since August (according to the date of the initial spec) and we'd been considering it for several months before that. Personally I've been hoping for this change for years. (And I'm not the only one.)

...

The main reason for the change was to correct what we believe was a mistake to change the reputation payout in the first place. That it also makes people feel better is something of a pleasant side-effect.

However, many users who saw that staff/moderator discussion talk as if this study was cited as a (or, the) motivating factor.

To me, it seems all these comments are consistent with the following theory:

• Many experienced members of SE staff, who are also long-standing users of the SE network, felt that halving rep for questions had not achieved its stated aims in somehow improving question quality, and should be reversed to encourage and facilitate involvement from users who write good-quality questions
• Senior decision-makers at director level, who seldom use the SE network and have much less experience with it, were not interested in prioritising this change...
• ...until they saw this study. After seeing this study, they gave this proposal the green light. This would fit Jon's timeline where it was under active consideration "several months" before August (i.e. from around March/ April?

If this is true, the study would be what got the proposal approved and signed off, but wasn't what motivated the proposal in the first place.

It would just be part of how they presented it to the boss in order to get it signed off (we've all been there!).

And it was part of how they presented it to moderators too, until they remembered that moderators do things like actually read studies, and notice if those studies presume gender based on usernames, exclude all users without gendered-real-name usernames, and presume that if a gender group that includes a higher % of early career developers asks more questions than a group that is on average older, it must be because of their gender somehow, not because people early in their careers inevitably have more questions.

---

If anyone has more insider knowledge or can view the moderator discussion and sees something that contradicts this theory, please comment.

Answer 2

When reading through the paper a few specific parts of it felt problematic for me

More specifically, we find significant gender gaps in activity: women are more likely to ask questions, while men provide more answers and cast more votes.

Which I guess could be an observation, but I didn't really find a great reason for the why and how we can help women actually fill that gap. Cause I'm pretty sure gender isn't a barrier to knowledge.

Using the Oaxaca-Blinder decomposition (Oaxaca 1973), a method from economics that to the best of our knowledge has not yet been previously applied to measure gender disparities in online communities, we decompose the outcome differences between men and women in terms of differences in their activity.

This bit is interesting. Unfortunately I'm not smart enough to actually get what Oaxaca-Blinder decomposition is, though since its novel to this application - I'm a little skeptical.

In the final part of the paper, we explore the consequences of a hypothetical redesign of the site’s reward system. The proposed alternative scoring system equalizes the rewards for well-liked questions and answers, a simple and justifiable change which does not penalize any group of users in absolute terms.

This sounds shiny.

Stack Overflow is itself a well-studied platform. Vasilescu et al. show that women are underrepresented in this community (Vasilescu et al. 2013). Interviews with a sample of Stack Overflow users highlight the barriers women have to greater participation. Women respondents listed the lack of awareness of some site features, the intimidating community size and their fear of lacking adequate qualifications as main barriers to participation (Ford et al. 2016).

Now this literally is a problem worth solving IMO. Why are these barriers there? How do we mitigate them? Getting engaged users is essential if you want a constant, vibrant representation of a specific segment of the community

Now the big problem I have with this paper

"We found that while the method performed very well on men (97% agreement with our manual check), our manual check agreed only in 44 out of 100 cases of women. This replicates the recent finding by Ford et al. (2017) that genderComputer sacrifices precision for greater recall when inferring women users."

They did run further rounds - but that also means that feels like selecting for women who use obviously female names, and men who use obviously male names. This basically doesn't select for anonymous users - and someone uncomfortable with being identified by their gender would do that or use an ambiguous name

Here's the frustrating part. I can't share more. Unlike our rogue moderator - I do respect the sanctity of folks trusting you not to share things.

Its worth considering, this wasn't in the final announcement. At best it was one of many things considered both in internal messaging and for external messaging.