8

This is just an idea I've been toying with , really. I'm curious about the correlation between question length and question quality. "Quality" here would be translated into "likelihood of not being closed."

Because when you go through Stack Overflow, most of the very short questions end up having issues of some kind.

But is there a way for us to measure whether there's a lower threshold whereby we are very certain that the question is problematic? For example, I doubt anyone would argue that you can make a great question with just 15 or so words.

For example, here's a demo I just ran & subsequently deleted:

enter image description here

But surely you'd rather see an example from the real jungle, here's a recent one:

enter image description here

My question is - why were the body-size limits determined to be 30 characters (though I believe there's a typo there, as it's not simply 30 char's)?

  • 1
    Yeah, not to mention you can still annoy a reasonable number of people before you delete it. And you have a real example anyway! – Blorgbeard is out Mar 14 '14 at 2:40
  • 1
    Also, are you aware of the data explorer? You could probably write a query to investigate this, if someone hasn't already. – Blorgbeard is out Mar 14 '14 at 2:41
  • 4
    Some of the worst questions that I have read have been very long. Some are huge code dumps, asking, "How can I make this faster?" – Matthew Lundberg Mar 14 '14 at 3:42
17
+50

I created this query:

SELECT Score AS PostScore, AVG(LEN(Body)) AS AvgLen
FROM Posts 
WHERE PostTypeId = 1 
  AND CreationDate > '20120101' 
  AND Score < 20 
  AND Score > -20
GROUP BY Score ORDER BY Score ASC

which returns the average post length for each score (-20 to 20 for relevant sample size). The query is here.

What it shows is the following:

enter image description here

It would appear that there is a critical post size around 1400 characters, after which the size of the post becomes somewhat irrelevant. There is too much noise from -19 to -10ish for any significant conclusions, but it does appear to also have a downward trend.

Thus, we can, with relative confidence, conclude that your hypothesis is correct. Shorter questions tend to receive more downvotes.

It is actually more significant for answers than it is for questions! With a simple modification (PostTypeId = 2), we can change the query so that it applies to answers instead of questions. Here is what we see:

enter image description here

There is quite clearly a significant correlation between answer length and vote count. Answers, I think, are probably more indicative that the system works than questions: good, detailed content is clearly receiving upvotes.

As opposed to questions, it actually appears as if the more you write, the better off your answer fares (i.e., for the most part, the more detailed and helpful your answer is, the more upvotes it receives). It also appears like it doesn't take much to improve your answer's score significantly.

  • 5
    I'm speechless, the graph & query is excellent , thank you so much!! – Adel Mar 14 '14 at 4:31
  • 1
    @Adel Thanks! :] Hopefully it helps. I've added some information about answers as well. – user206222 Mar 14 '14 at 4:38
  • Good to see more-related query than mine! Mine was biased enough since I specified the score, that's why I couldn't make any conclusion afterward. However, for answer, I certainly agree that the more descriptive it is, the more helpful! – Meta Andrew T. Mar 14 '14 at 4:43
  • If you expand the search out to questions of a score of 100 or more, and include the zero-score questions, the median length is actually 1,286.61. Without the zero-score questions, the median length is 1,285.45. Not a whole lot of difference. There's also the case that negative scored questions (down to -38) have an average length of 974.27. – Makoto Mar 14 '14 at 5:16
  • @Makoto Take a look at the number of posts you're averaging, though - the sample size really isn't all that high. 1-20 posts isn't nearly enough to cover an anomaly - you can see the query starts to have problems even around -10 or 11. I limited it to 20 for sample size. – user206222 Mar 14 '14 at 5:44
  • 1
    query results for questions are skewed because it misses deleted questions. To estimate the impact of roomba auto-deletion, you can query the "time window" when it did not kick in yet, that is last 9 days off the data snapshot date – gnat Mar 14 '14 at 8:00
  • 1
    One reason is because long answers tend to have detailed explanations. Long questions tend to have big blocks of copy-pasted code... – Tim B Mar 14 '14 at 10:18
  • @gnat Since the script is with respect to score, we can probably assume this is still a relatively accurate sample of downvoted questions. – user206222 Mar 14 '14 at 16:42
  • 1
    well, you can test this assumption by comparing full timescale results to those obtained with no-roomba-window (I do not make any assumptions here, merely point that this is somewhat verifiable thing) – gnat Mar 14 '14 at 16:44
  • 2
    @gnat Yeah, that is true. It's a valid question! Just a little disinclined to look into it at the moment. I may come back to this and do a more extensive analysis. – user206222 Mar 14 '14 at 16:45
  • I think a more useful graph would have question length of the x axis (on a log scale) and average score on the y axis. I have no idea how to make that kind of query though. – curiousdannii Mar 24 '14 at 5:00
4

Here is a query for Data Explorer that I use to explore this question

SELECT TOP 10 Id, Title, Body, Score
FROM Posts
WHERE LEN(Body) < 50 AND
      Score > 10 AND
      PostTypeId = 1 AND
      CreationDate > '20120101'
ORDER BY Score DESC

Link: https://data.stackexchange.com/stackoverflow/query/174387

This assumes that

  • The question has less than 50 characters
  • It has score more than 10 (deemed useful)
  • It's posted after January 1st, 2012 (to avoid difference with SO's recent rules)

As of current, it returns 3 questions:

From what I see, it seems that OPs didn't do any effort at all, however it's deemed useful by communities. (Almost) no downvote, and not closed! (though I might downvote them, but I didn't find any reason to flag them)


Edit: On the contrary, I made small modification to the query to show how lengthy questions might be bad.

SELECT TOP 10 Id, Title, Body, Score
FROM Posts
WHERE LEN(Body) > 5000 AND
      Score < -5 AND
      PostTypeId = 1 AND
      CreationDate > '20120101'
ORDER BY Score

Link: https://data.stackexchange.com/stackoverflow/query/174387

This instead shows questions with more than 5000 characters and have score less than -5. Samples:

Most of them are just a big code dump, are unclear and should be closed.

  • Thanks so much! I will be playing with that Data Explorer, it's quite telling. – Adel Mar 14 '14 at 3:52
3

Well, this isn't exactly what you asked, but there's an article here about the characteristics of a good SO question, based on queries in the data explorer. The criteria they used for determining the quality of a question wasn't your "likelihood of not being closed", but whether it gets good (upvoted) answers quickly.

They claim that shorter answers are more likely to get useful answers. Of course, by short they don't mean as short as your one liner examples.

Probably the strongest pattern we saw across all the different queries and scripts we ran was this: the shorter the question, the more likely you are to get a quick and useful answer. The average length of a ‘good’ question was around 1200 characters (~3-4 short paragraphs), while the average length of a standard question was 1800 characters. The strongest effect of the length was on the view count and votes, where the average length of the top questions was around 50% of that of standard questions.

There’s no such thing as too short, either — really short questions (around 200-300 characters) received the best results.

It's an interesting read.

  • Wow, that's really interesting. quite counterintuitive also, hmm so generally speaking shorter is better. – Adel Mar 14 '14 at 3:35
  • 3
    It is not counter-intuitive. Short and to the point is best. – Matthew Lundberg Mar 14 '14 at 3:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .