22

Today I created a query to plot the average answer score by answer length. The results show that:

  • really short answers (less than 30 or 40 characters) often score quite well,
  • write beyond that and you get about 9 rep for each 1k characters you put into an answer, and
  • beyond about 2k characters you enter a realm of high volatility.

Average Stack Overflow Answer Score by Answer Length

What is it about the QA format of Stack Overflow, or the sorts of questions and answers asked here, that 'discourages' longer answers, putting them into the nature of fickle voting?

| improve this question | |
  • 28
    My guess is your data is skewed because the frequency of high character answers is far lower. – jjnguy Aug 26 '11 at 22:11
  • 3
    Bingo. If you draw a line through the center of that mess you get a perfectly straight trend line. – user102937 Aug 26 '11 at 22:13
  • 1
    @jinguy It is lower, but I'm not sure about 'far lower'. There are 17.4k answers with character length between 2k and 2.1k, and 4.6k answers between 3k and 3.1 length. So yes, there's less than 1/3...but not something like 1/10th or 1/100th. Further: over 4,000 answers in that region is a pretty large sample size (I'd think) providing smoother averages. – Phrogz Aug 26 '11 at 22:38
  • 2
    Could you plot a second graph with the number of answers in each chunk? – Paŭlo Ebermann Aug 26 '11 at 22:44
  • (Also, it is not 0.9 rep per 1000 characters, but 0.9 upvotes, I think.) – Paŭlo Ebermann Aug 26 '11 at 22:46
  • The spike at "really short" answers is likely since bad too short ones are often deleted as not an answer (or converted to comments). – Paŭlo Ebermann Aug 26 '11 at 22:48
  • 3
    In addition to the average answer score, can you please also plot the median answer score? Bill really has a point in his answer with "A few very highly-voted answers in a bucket that size can cause it to spike significantly." – Hendrik Vogt Aug 27 '11 at 7:09
  • 2
    Off topic: there's a creepy statistic in this graph. Doubling the answer score requires four times the effort. Clearly it pays off to only post short answers. Something wrong with that. – Uphill Luge Sep 3 '11 at 17:04
26

beyond about 2k characters you enter a realm of high volatility.

I don't think that's accurate. I see the volatility steadily increasing throughout the curve, starting at around 200 or so characters (I'm running your query with a chunk size of 10).

The increased volatility can be explained by the decrease in frequency of high-character answers, as @jjnguy mentioned in the comments. Looking around the 180 - 220 character range, each bucket has about 70,000 answers, so the average score will be skewed very little by outliers. In the 3600 character range though, each bucket only has about 250 answers. A few very highly-voted answers in a bucket that size can cause it to spike significantly.

| improve this answer | |
  • 1
    Kinda like saying: (1) more observations = less variance in the mean (2) more observations = less sensitive to outliers. – M. Tibbits Aug 27 '11 at 3:30
  • Indeed not accurate, the time spent on a longer answer could also be spent on multiple shorter answers. – Tamara Wijsman Aug 28 '11 at 1:40
  • Perhaps a plot of the standard deviation for each bucket as a function of character count would be more appropriate here, along with a widening of the buckets. I don't see how a visual inspection of the averages for each character count gives us any real indication of variability. – Brad Larson Aug 28 '11 at 1:41
  • @Brad: Since we have the raw data available, it would definitely make sense to measure the variance directly instead of inferring it from the averages. – Bill the Lizard Aug 28 '11 at 3:43
5

It may be that fewer people actually read the long answers carefully, so they just vote according to how people before them voted, amplifying trends. (I don't think the data dump has the info needed to investigate voting order, but the team probably could if they cared)

Or, it could be because long answers often start as short answers, but the late edits don't get as much attention so the voting is more sporadic. (Could be investigated by adjusting for edit counts or times)

Or, it could be a result of copy-pasta from common documentation which could lead to many similar-scoring answers in the same size range. (Could investigate with diffs if you're really hard core)

Or, it may be telling us that after 3000 characters additional length isn't always beneficial. At some point thorough answers become overly verbose. There's value in making your point concisely as well. (I'm not sure how you'd investigate this other than ruling out other possibilities)

| improve this answer | |
3

I would guess that shorter answers tend to just answer the question. A bit of explaining (more length) will get you more votes, which makes sense.

Longer answers, though, generally go far beyond a simple answer. They might touch on a variety of related topics, or explain the rationale for why your "bug" is , and so on. Some of these are just long, wasting your time with extra information: more downvotes. Others are fascinating and truly help your understanding, and garner tons of upvotes. Few of these long answers simply answer the question with nothing extra and receive a middling score.

That's my theory, anyways. Plus a touch of the skew mentioned in the comments.

| improve this answer | |
1

The longer an answer is - the more each voter has to either to like or dislike about the answer. It's not hard to see how a very long answer is more likely to polarize a population of voters. "Useful" implies properly edited and targeted for the reader. It shouldn't necessarily be the fault of the answer if the voter is reading the wrong question - but I can see how that would skew the votes in the described manner.

So it goes in politics, in personal relationships, and here. We give the benefit of the doubt when we don't have all the facts. A long answer gives us those "facts".

It would be very interesting to see the voting rate on these longer questions, as well as some idea how wide or thing the standard deviation is. Averages and undefined sorting bins are very poor levers for analysis - it's not clear that the graph you have isn't a simply measurement noise from a long tail data set.

| improve this answer | |

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .