I recently asked a SO question about language productivity.
What emerged from the question was that there really was not much good research into the topic of language productivity and that any research which had been done was from the 2000 time-frame. What makes research like this difficult is finding a large sample size and finding problems which don't favour one class of language greatly over another. That got me thinking, what better source of programmers is there than stackoverflow? There are developers from all across the spectrum of languages and abilities; there is even a pretty good geographic disbursement.
Let's do this research ourselves! I propose a stackoverflow language programming contest. We'll develop a suite of programming tasks which try as hard as possible to not focus on the advantages of one particular language and gather metrics. I think we should gather
- Time taken to develop
- Lines of code required
- Runtime over the same input
- Memory usage over the same input
- Experience of the programmer (years or some other metric, KLOCs written? Degrees?)
- Other things I haven't thought of
I'll set up a site to gather people's solutions to the problems and collate statistics but the problems should be proposed by the community. We'll allow people to checkout the problem set, time how long it takes to the to complete it and then submit the code for their answers. I'll run the code and benchmark the results and after, say two weeks of having the contest open, publish my results as well as the dataset for anybody else to analyze.
What do people think? Would any of you participate? What questions do you propose? Can you think of other metrics? Which languages? I would think at least