7

Context

Developers responsible for the [your favourite programming language] (YFPL) component in your software system have just left the project. How difficult is it to replace them by other developers from within the same project? In other words, how risky is YFPL within your project?

Cases:

  • some YFPL developers are still active: no problem, just reuse some of them;
  • all YFPL developers have left: hire more YFPL developers / wait for new YFPL developers to join, not applicable;
  • all YFPL developers have left: use developers from within the same project, who have not worked on YFPL, and may or may not have knowledge of YFPL. Hence, how likely is it that someone who knows [your second-favourite programming language] (YSFPL) also knows YFPL?

Approach

In my current research I'm trying to answer the above question by looking at SO questions and answers, to provide me with a general view of what programming languages developers commonly master together. The approach is based on frequent itemset mining, commonly used in supermarket research (people who buy diapers also buy beer).

As result one could see, e.g., that is is common for developers that answer YSFPL to also answer YFPL (be it the same question containing both tags, or different questions alltogether), but it is very uncommon for developers that answer YTFPL (T for third) to also answer YFPL questions.

I would conclude from this that it is difficult for YTFPL developers, but easy for YSFPL developers, to take over the YFPL component.

Question

How valid are my assumptions?

  • the SO community is representative;
  • the tags a SO user collects as a result of asking/answering questions are representative of her knowledge (also mentioned here).
5
  • Why not hit it with a science hammer? There are statistical techniques that will allow you to estimate the confidence level of your data, such as Standard Deviation. There's also dozens of ways you can slice and dice this data (upvotes per post, for example) to improve the confidence level. The best correlation, unfortunately, will occur with the fewest members (the high-rep ones).
    – user102937
    Apr 4, 2012 at 15:01
  • Statistics.SE can help you with stuff like this. Apr 4, 2012 at 15:09
  • I'm not excluding statistics, I was just trying to be brief in the question. I think statistics and ARM (association rule mining) are complementary. I am reluctant to filter on rep because rep is a measure of involvement rather than expertise. I am however keen to find good ways to filter out "noise", keeping in mind that the more complex the filtering is, the more complaints the reviewers of the paper will probably have :P Apr 4, 2012 at 15:16
  • 1
    "rep is a measure of involvement rather than expertise", ooh, burn :p
    – AakashM
    Apr 4, 2012 at 15:32
  • It's difficult to argue in a scientific paper why you chose to ignore people with rep < 1234. People will immediately complain "Why 1234 and not 12345?" More advanced things, like 80% of the total rep is "owned" by 20% of the people, will not do the trick either. Apr 4, 2012 at 15:42

1 Answer 1

5

To the validity of your assumptions:

the SO community is representative

The SO community is representative of something, but it likely isn't representative of all programmers everywhere, especially when you begin constraining what "the SO community" means. This comes from a number of different factors:

  1. SO is an English speaking site.
    While it seems to be true that English is the lingua franca of programming, there are a large number of programmers out there who find asking and answering questions in English to be a major barrier and therefore don't participate on SO or don't participate as much as they might otherwise. This will tend to skew your results.

  2. Not all programming technologies are equally represented on SO.
    Many technologies that were established well before SO even existed already had other support mechanisms in place, like email lists, and to a large extent these other mechanisms are still used to handle questions about those technologies.

    Another element of this is the "positive feedback loop" effect that is created by more and better questions for a technology being created on SO. As more people link to SO questions, SO gets ranked higher and higher in search results for a particular technology, driving in more users (and content generators) for that technology, which just strengthens that technology in the SO ecosystem, possibly disproportionately to the actual use of that technology. There are a number of other factors that also feed into the feedback loop, but the net effect of swelling a technology, regardless of actual usage, is still there.

  3. The more someone cares about a community, the more involved they will be.
    The reverse of this statement is also true, and it essentially means that SO involvement is much more likely to come from the top 50% who most care about programming than it is from the bottom 50%. To put it another way, the SO population is going to tend to be skewed towards people who care more than towards people who care less.

the tags a SO user collects as a result of asking/answering questions are representative of her knowledge

I would agree that the tags on a question to which a user provides an answer are more likely to align with that user's knowledge base, but I don't think the tags on the questions a user asks correlates with their knowledge in those areas. Instead, I would posit that the tags on questions a user asks relate to their interests and/or activities.*

The "peer pressure" (and rep pressure) to provide a good answer, rather than a bad one, will tend to discourage those with limited or what they perceive to be lesser knowledge in an area from answering a question. At the same time, the "fastest gun in the west" effect means that a user with some knowledge of a topic may decide not to look at or answer a question simply because someone else already has given an answer. There are likely a large number of questions that many people could answer equally well, but most people chose not to because someone had already answered the question completely.

Both of these effects will tend to skew the representation of what a user "knows" based on answers to questions in a tag simply due to competition. Again, it's a case of those that are most involved being "promoted" over those that are less involved.

In order for any of this to be accurate, however, the base assumption that you have to have is that tags on questions are accurate. I think most tags are fairly accurate, but there is definitely a substantial amount of fuzziness there.

* Yes, long-term interests/activities are likely to impart knowledge, but it can't be used to measure current knowledge

1
  • +1 for "a user with some knowledge of a topic may decide not to look at or answer a question..." I spend about 40% of my time coding in Python but there are so many, excellent, quick answering python coders on SO that there's very little point answering questions. Apr 4, 2012 at 18:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .