What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

I recently asked a SO question about language productivity.

What emerged from the question was that there really was not much good research into the topic of language productivity and that any research which had been done was from the 2000 time-frame. What makes research like this difficult is finding a large sample size and finding problems which don't favour one class of language greatly over another. That got me thinking, what better source of programmers is there than stackoverflow? There are developers from all across the spectrum of languages and abilities; there is even a pretty good geographic disbursement.

Let's do this research ourselves! I propose a stackoverflow language programming contest. We'll develop a suite of programming tasks which try as hard as possible to not focus on the advantages of one particular language and gather metrics. I think we should gather

  • Time taken to develop
  • Lines of code required
  • Runtime over the same input
  • Memory usage over the same input
  • Experience of the programmer (years or some other metric, KLOCs written? Degrees?)
  • Other things I haven't thought of

I'll set up a site to gather people's solutions to the problems and collate statistics but the problems should be proposed by the community. We'll allow people to checkout the problem set, time how long it takes to the to complete it and then submit the code for their answers. I'll run the code and benchmark the results and after, say two weeks of having the contest open, publish my results as well as the dataset for anybody else to analyze.

What do people think? Would any of you participate? What questions do you propose? Can you think of other metrics? Which languages? I would think at least

  • Java
  • C
  • C#
  • Ruby
  • Perl
  • Python
share|improve this question

3 Answers 3

What makes research like this difficult is finding a large sample size and finding problems which don't favour one class of language greatly over another. That got me thinking, what better source of programmers is there than stackoverflow?

StackOverflow may have a large sample size, but it's a poor choice for serious research because:

  • Research (poll-type) questions are not the purpose of the site, and even if you get massive approval on meta, that doesn't mean it will be well-received on SO proper.
  • The languages are not equally represented in terms of number of participants on SO.
  • The languages are not equally represented in terms of the skill level of participants on SO.
  • The questions would necessarily be opt-in. Those people who produce poor results may elect not to return their data, tainting the overall dataset.
  • Properly measuring effort is incredibly difficult. Do you expect people to drop everything and work on these questions 100%?

As you've probably seen in your research, these types of studies are best done with a controlled sample of individuals from industry under controlled circumstances. I.e., they are given identical resources and expected to work on the challenges given exclusively for as long as it takes. Typically a few consecutive days are reserved so that everyone has time to finish a decent-sized challenge. Studies can be done more informally, but only at the cost of the quality of the results.

This is the reason the sample sizes are usually small: It's desirable to control as many factors as possible, which is difficult with a large number of participants. After all, you can't expect 10% of the industry to take a week-long vacation just to get you some research data.

Getting a large sample size while controlling the experiment to the degree necessary is a very ambitious and worthy goal, and if you can do it, there's probably a big fat grant just waiting to fund it. But I don't think StackOverflow is going to get you there.

share|improve this answer
    
I'm not suggesting that this be a feature of SO but just that we use SO as the source of participants. –  stimms Dec 13 '09 at 18:43

TopCoder has been running contests like this for years. But I don't know if they've ever done a study comparing the results. The Stack Overflow system might present a bigger challenge when it comes to measuring results, as keeping existing solutions from late participants would be tricky...

I strongly suspect that using such results would be tricky... Personally, I spend far more time researching a new project than I actually do coding up a solution, regardless of which language I end up using for it. Also, learning to use an unfamiliar or poorly-designed API can chew up vast amounts of time regardless of the language used to call it. Would you present contestants with a fully-developed algorithm and ask only that they implement it? How would you account for language bias in the algorithm description?

That said, it might be fun to try (if not particularly useful). Perhaps a variation (a lowest time-to-code goal instead of or in addition to the lowest LoC goal) of the existing Code Golf contests would be a good place to start?

share|improve this answer

The difficulty with scientific studies is controlling all the variables.

This one has too many to be controlled, and relying of sheer numbers to average out the influence of the variables would produce meaningless results.

Most development projects are unrepeatable experiments, because the participants learn as they go. To run an accurate scientific experiment you'd have to rewind time, erase everyone's memory, and run the project again using a different language.

and of course, that's un-possible ;-)

share|improve this answer
    
There's a lot of scientific experiments like this. All the complications (including the learning effect) can be handled, more or less, statistically. Of course, that requires a large sample size and a good deal of statistical sophistication on the part of experimenters and reviewers. –  David Thornley Dec 14 '09 at 18:59
    
@[David Thornley]: interesting - got links? –  Steven A. Lowe Dec 15 '09 at 1:04
    
@[Steven A. Lowe]. Check out a journal called "Empirical Software Engineering", and a conference called "Empirical Software Engineering and Measurement (ESEM)". –  Lorin Hochstein Jan 30 '10 at 22:33

You must log in to answer this question.