I'm doing some quantitative research on SO and I'm trying to find good "natural" or quasi-experiments that have occurred or will occur in SO.

What I mean by a natural experiment is some policy change, events or even feature of the system that manipulates things like:

1) Views received
2) Answers received
3) Probability that an answer is accepted
4) Up votes / down votes

The changes should manipulate these variables in a manner that's similar to what you would find in a true experiment.

To provide an example, imagine I wanted to see if users asked more questions later if their first question received many answers. You obviously cannot just treat "answers received on first question" as an independent variable---it's almost certainly correlated with the posters characteristics, which will affect both the answers they receive and their future behavior. However, suppose by chance that a posters' question got a 10X increase in views because X happened that day, then we could exploit this shock to try to uncover the causal effect of getting more views by comparing them to others with ~X.

So, to summarize, I'm looking for some ideas on X.

Also - it would be helpful if you have an approximate date when the change was implemented or will be implemented.

Thanks for your answers - these are great. Too bad I had to pick just one answer. "Gentlemen, you've both worked very hard, and in a way, you're both winners. But in another more accurate way, Barney is the winner."

  • I'm looking to do a similar analysis to @john-horton, and these answers are great, but I'm curious whether there's anything more recent than three years ago. Thanks!
    – Noleli
    May 19 '13 at 21:10

Change: Order answers with the same vote total randomly rather than based on time.
Implemented: circa Aug 2009 (see here)
Result: Theoretically, to level the playing field with respect to time of arrival as a solution to the FGITW problem.

  • Thanks - do you know about when this happened? I've been searching but haven't found a good link yet. Mar 16 '10 at 19:45
  • @John -- updated with date.
    – tvanfosson
    Mar 16 '10 at 19:58

Change: Display accepted answer ratio.
Result: A lot of answers being accepted immediately after the policy change, and now some people are unwilling to answer questions from users with a low ratio.

Change: Electorate badge for voting on a lot of questions as well as answers.
Result: I've heard of at least one user now voting more on questions; I don't know what effect it's had globally though.

  • 1
    Negative effect: I don't vote on answers at all because of that badge
    – juan
    Mar 16 '10 at 14:15
  • 2
    @Downvoter -- not even downvotes. Now that's a shame. ;-)
    – tvanfosson
    Mar 16 '10 at 14:28
  • Aww, man, I thought everyone had only posted one change per answer. I'd just finished writing up my analysis of the electorate badge when I chanced to scroll up one last time and noticed the second change.
    – mmyers
    Mar 16 '10 at 14:28
  • 1
    I definitely increased my voting on questions when the electorate badge came out, and even after receiving it, have kept up the habit. Mar 16 '10 at 15:19
  • 3
    "Display accepted answer ration" certainly has increased the amount of complaining, about users who rarely accept answers. Mar 16 '10 at 15:37
  • 1
    After the Electorate badge appeared my answer voting stayed pretty much the same, but my question voting (specifically downvotes) spiked dramatically. Now that I have the badge, I still downvote off-topic questions more often than I did before, but not as much as I was when the badge first appeared. Mar 16 '10 at 20:20

Change: You are unable to change your display name more than once per month
Result: I cannot find a suitable display name I'd like to have for a month so I keep the one I had when the change was implemented, also, it made Meta more boring.

  • 1
    Screw you, pal.
    – Welbog
    Mar 16 '10 at 15:47
  • 1
    Yeah, more boring sucks!
    – Ether
    Mar 16 '10 at 16:16

Change: Low rep users get a "have you considered commenting" popup when they down-vote an answer.
Implemented: July 2009 (< 500 rep), modified December 2009 (< 2000 rep) - see the revision history for Jeff's answer that announced the change.
Result: Again theoretically, increase the number of comments that go with down-votes so the poster can know what was wrong with the question or answer and so correct it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .