Why is there such score volatility above 2k character answers?

Question

Today I created a query to plot the average answer score by answer length. The results show that:

• really short answers (less than 30 or 40 characters) often score quite well,
• write beyond that and you get about 9 rep for each 1k characters you put into an answer, and
• beyond about 2k characters you enter a realm of high volatility.

What is it about the QA format of Stack Overflow, or the sorts of questions and answers asked here, that 'discourages' longer answers, putting them into the nature of fickle voting?

Answer 1

beyond about 2k characters you enter a realm of high volatility.

I don't think that's accurate. I see the volatility steadily increasing throughout the curve, starting at around 200 or so characters (I'm running your query with a chunk size of 10).

The increased volatility can be explained by the decrease in frequency of high-character answers, as @jjnguy mentioned in the comments. Looking around the 180 - 220 character range, each bucket has about 70,000 answers, so the average score will be skewed very little by outliers. In the 3600 character range though, each bucket only has about 250 answers. A few very highly-voted answers in a bucket that size can cause it to spike significantly.

Answer 2

It may be that fewer people actually read the long answers carefully, so they just vote according to how people before them voted, amplifying trends. (I don't think the data dump has the info needed to investigate voting order, but the team probably could if they cared)

Or, it could be because long answers often start as short answers, but the late edits don't get as much attention so the voting is more sporadic. (Could be investigated by adjusting for edit counts or times)

Or, it could be a result of copy-pasta from common documentation which could lead to many similar-scoring answers in the same size range. (Could investigate with diffs if you're really hard core)

Or, it may be telling us that after 3000 characters additional length isn't always beneficial. At some point thorough answers become overly verbose. There's value in making your point concisely as well. (I'm not sure how you'd investigate this other than ruling out other possibilities)

Answer 3

I would guess that shorter answers tend to just answer the question. A bit of explaining (more length) will get you more votes, which makes sense.

Longer answers, though, generally go far beyond a simple answer. They might touch on a variety of related topics, or explain the rationale for why your "bug" is status-bydesign, and so on. Some of these are just long, wasting your time with extra information: more downvotes. Others are fascinating and truly help your understanding, and garner tons of upvotes. Few of these long answers simply answer the question with nothing extra and receive a middling score.

That's my theory, anyways. Plus a touch of the skew mentioned in the comments.

Answer 4

The longer an answer is - the more each voter has to either to like or dislike about the answer. It's not hard to see how a very long answer is more likely to polarize a population of voters. "Useful" implies properly edited and targeted for the reader. It shouldn't necessarily be the fault of the answer if the voter is reading the wrong question - but I can see how that would skew the votes in the described manner.

So it goes in politics, in personal relationships, and here. We give the benefit of the doubt when we don't have all the facts. A long answer gives us those "facts".

It would be very interesting to see the voting rate on these longer questions, as well as some idea how wide or thing the standard deviation is. Averages and undefined sorting bins are very poor levers for analysis - it's not clear that the graph you have isn't a simply measurement noise from a long tail data set.