I'm doing some quantitative research on SO and I'm trying to find good "natural" or quasi-experiments that have occurred or will occur in SO.

What I mean by a natural experiment is some policy change, events or even feature of the system that manipulates things like:

1) Views received
2) Answers received
3) Probability that an answer is accepted
4) Up votes / down votes

The changes should manipulate these variables in a manner that's similar to what you would find in a true experiment.

To provide an example, imagine I wanted to see if users asked more questions later if their first question received many answers. You obviously cannot just treat "answers received on first question" as an independent variable---it's almost certainly correlated with the posters characteristics, which will affect both the answers they receive and their future behavior. However, suppose by chance that a posters' question got a 10X increase in views because X happened that day, then we could exploit this shock to try to uncover the causal effect of getting more views by comparing them to others with ~X.

So, to summarize, I'm looking for some ideas on X.

Also - it would be helpful if you have an approximate date when the change was implemented or will be implemented.

Thanks for your answers - these are great. Too bad I had to pick just one answer. "Gentlemen, you've both worked very hard, and in a way, you're both winners. But in another more accurate way, Barney is the winner."

share|improve this question
I'm looking to do a similar analysis to @john-horton, and these answers are great, but I'm curious whether there's anything more recent than three years ago. Thanks! –  Noleli May 19 '13 at 21:10
add comment

4 Answers

up vote 5 down vote accepted

Change: Order answers with the same vote total randomly rather than based on time.
Implemented: circa Aug 2009 (see here)
Result: Theoretically, to level the playing field with respect to time of arrival as a solution to the FGITW problem.

share|improve this answer
Thanks - do you know about when this happened? I've been searching but haven't found a good link yet. –  John Horton Mar 16 '10 at 19:45
@John -- updated with date. –  tvanfosson Mar 16 '10 at 19:58
add comment

Change: Display accepted answer ratio.
Result: A lot of answers being accepted immediately after the policy change, and now some people are unwilling to answer questions from users with a low ratio.

Change: Electorate badge for voting on a lot of questions as well as answers.
Result: I've heard of at least one user now voting more on questions; I don't know what effect it's had globally though.

share|improve this answer
Negative effect: I don't vote on answers at all because of that badge –  jmfsg Mar 16 '10 at 14:15
@Downvoter -- not even downvotes. Now that's a shame. ;-) –  tvanfosson Mar 16 '10 at 14:28
Aww, man, I thought everyone had only posted one change per answer. I'd just finished writing up my analysis of the electorate badge when I chanced to scroll up one last time and noticed the second change. –  mmyers Mar 16 '10 at 14:28
I definitely increased my voting on questions when the electorate badge came out, and even after receiving it, have kept up the habit. –  Lance Roberts Mar 16 '10 at 15:19
"Display accepted answer ration" certainly has increased the amount of complaining, about users who rarely accept answers. –  Brad Gilbert Mar 16 '10 at 15:37
After the Electorate badge appeared my answer voting stayed pretty much the same, but my question voting (specifically downvotes) spiked dramatically. Now that I have the badge, I still downvote off-topic questions more often than I did before, but not as much as I was when the badge first appeared. –  gnostradamus Mar 16 '10 at 20:20
add comment

Change: You are unable to change your display name more than once per month
Result: I cannot find a suitable display name I'd like to have for a month so I keep the one I had when the change was implemented, also, it made Meta more boring.

share|improve this answer
Screw you, pal. –  XMLbog Mar 16 '10 at 15:47
Yeah, more boring sucks! –  Ether Mar 16 '10 at 16:16
add comment

Change: Low rep users get a "have you considered commenting" popup when they down-vote an answer.
Implemented: July 2009 (< 500 rep), modified December 2009 (< 2000 rep) - see the revision history for Jeff's answer that announced the change.
Result: Again theoretically, increase the number of comments that go with down-votes so the poster can know what was wrong with the question or answer and so correct it.

share|improve this answer
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .