As sites grow, there's a noticeable increase in community unhappiness with question quality. Quite a bit of the problem is simply a matter of scale, but sites sometimes optimize for extremes. Bad askers don't notice or care that their questions are bad and potential good askers tend to be more sensitive to criticism. A dangerous feedback cycle sets in where members of the site are rightly critical of new questions which discourages potentially good askers from asking which leaves a greater percentage of bad questions remaining.

We've heard lots of suggestions and we'd like to start testing some of them. But in order to do that, we need an objective measure of question quality. Or rather, we need an objective measure of the community perception of question quality, which is thankfully easier to obtain. As Clay Shirky says:

So the group is real. It will exhibit emergent effects. It can't be ignored, and it can't be programmed, which means you have an ongoing issue. And the best pattern, or at least the pattern that's worked the most often, is to put into the hands of the group itself the responsibility for defining what value is, and defending that value, rather than trying to ascribe those things in the software upfront.

Taking that to heart, we've defined a simple "question grade" based on objective measures of community feedback:

  • Good

    (Score > 0 or (AnswerCount > 0 and Score = 0)) and !Closed 
  • Bad

    Score < 0 or Closed
  • Neutral

     Score = 0 and !Closed and AnswerCount = 0

In all cases, Closed excludes duplicate closures because not all duplicates are bad for the site. Duplicate questions are a big part of community frustration, so it feels weird to ignore them. Fortunately, unhelpful duplicates tend to be downvoted as well as closed. When we run tests of features specifically designed to reduce duplicates (such as search improvements), we'll use a metric that takes duplicate closure into account.

Accepted answers aren't useful in this case since we are measuring the community's assessment rather than asker's. This is also a potential reason for excluding duplicate closures as askers have a say in that case.

36% of deletions within the first three days of a questions life are executed by the asker. Beyond that time, most deletions are automatic and intended to remove cruft and not necessarily because questions are bad. We considered counting just community-initiated deletions, but that made question grade calculations noticeably more expensive to calculate. Fortunately, most deleted questions are downvoted or closed, so statistically speaking, deletion adds very little signal.

Finally, the grade is calculated after 3 days. Our data team discovered that on Stack Overflow:

  • 18.5% of posts never change their status: that is, they’re permanently neutral.
  • 67.6% of posts have “solidified” their status in the first hour.
  • 91.6% solidify within 3 days.

Solidified quality

For the purpose of question quality A/B tests, we're going to consider neutral questions as equivalent to bad questions. Since that divides questions roughly in half on Stack Overflow, tests will resolve more quickly. Our data team also reports that using this criteria, 73% of questions solidify into their good/not-good category within an hour.

Validation of question grades

For the purposes of A/B testing, we need a measure that quickly categorizes all questions. There are certainly going to be edge cases, such as boring questions that happen to get amazing answers or decent questions that are ignored for some reason. To quote Carveth Read:

It is better to be vaguely right than exactly wrong.

This very simple grading of questions matches the intuition most of us have that as a percentage of all questions, fewer of them are good or even answerable. Look no further than the steady decline of good questions on Stack Overflow:

Percentage of asked questions by grade on Stack Overflow

You can even spot the noticeable dip of 2014 which coincided with the observation that Stack Overflow had become particularly negative. So while this metric is not perfect, it is simple and seems to accurately track the community's evaluation of new questions as they come in. When tackling technical solutions to question quality, we can be confident we're on the right track if there's a statistically meaningful increase in "good" questions as defined above.

If you are interested in how other communities have evaluated their questions over time, you can use this query in the Stack Exchange Data Explorer. There's a very obvious trend of decreasing question grade as sites grow. While our current focus is on Stack Overflow because of its scale and importance to our business, the testing we do there will be useful for all other Stack Exchange sites that grow large enough. That will likely include sites customers pay for. Whether your site has a question quality problem now or will in the future, we'd like to verify that changes we make a positive difference.

  • is "answers" in your formulas number of answers, or scores of answers, or what? – Servy Nov 7 '17 at 18:38
  • "You can even spot the noticeable dip of 2014 which conincided with the observation that Stack Overflow had become particularly negative." People have been posting questions on meta complaining about negativity constantly since it was created. If there was a dip at any point in time you could find a meta post for that same time period complaining about negativity. – Servy Nov 7 '17 at 18:43
  • 4
    Most are not quite so heavily referenced, @servy. The big event there wasn't the post, it was several months of plummeting quality, commemorated in several well-known meta threads. – Shog9 Nov 7 '17 at 18:46
  • 1
    @Servy: Sorry. Answer count. If a question hasn't been voted up or down, but does have an answer, it's "good" in this system. So many questions are not voted on at all, we needed some other signal. It's not perfect (someone might drop a joke answer that gets downvoted, for instance), but it better than just using question score. – Jon Ericson Nov 7 '17 at 18:47
  • @JonEricson Would using positively scored answers instead of just "has answers" be feasible? – Servy Nov 7 '17 at 18:48
  • @Servy Certainly possible, but not sure it is important. It's worth noting that we erred on keeping this as simple as possible. – Joe Friend Nov 7 '17 at 18:56
  • How is a question that has an answer but has no votes itself good? A question that isn’t considered to be helpful by the community but answered doesn’t seem all that amazing to me. What other types of questions are considered good? – Ramhound Nov 8 '17 at 3:34
  • 2
    @Ramhound: We aren't talking "amazing" just good enough to be answered and not get downvoted. Remember that question downvotes are free, so there's not reason not to do that if the question is bad. We're trying to get some signal from the huge number of questions that aren't voted on at all. – Jon Ericson Nov 8 '17 at 3:45
  • 5
    Have you considered plotting the question quality not as a function of time, but as a function of total amount of questions? That might make comparison of various sites easier and more fair (especially considering various growth rates, since time is removed from the equation). It might also shed light on optimal size of sites, thresholds for certain actions/campains, etc, especially if the trend is not linear and if it turns out to be similar for all sites. Just my $0.02. – Marten Koetsier Nov 22 '17 at 20:35
  • 3
    While you said you want to keep this measure simple, wouldn't you want Score = 0 and AnswerCount = 1 and SelfAnswered to fall into the Neutral bucket? That's a question that no-one else has done anything positive with. – Josh Caswell Nov 25 '17 at 14:50
  • 3
    This trend could also be attributed to quality expectations increasing vastly as SO has grown. No more 800-score questions asking to convince the asker that PHP is good, for example. – TylerH Dec 5 '17 at 19:31
  • 1
    Are deleted questions included in the quality/time graph? Bad questions may be deleted the longer they hang around, no? – Michael Feb 7 '18 at 14:14
  • 3
    @Michael: Yes, deleted questions are included in the query. Bad questions are, indeed, likely to get deleted. – Jon Ericson Feb 7 '18 at 18:25
  • Which analysis? The template results? At any rate, here's the graph of how many first votes came at each year interval for questions. I'm not sure where the 5 votes at 10 years comes from, but I'm guessing it's rounding? When you add in answering (not shown) there just aren't many questions that switch from neutral to good years down the line. 3 days captures most of the signal and is a lot more practical than waiting several years to evaluate an experiment. ;-) – Jon Ericson Feb 12 '18 at 23:44
  • 1
    @Rob: I wouldn't think of "Neutral" as definitive state. Instead, we don't have any signal to know if the question is good or not. Most close votes are definitive signals that the question is just not helpful on any site. Even so, the difference between including it and not is marginal as your query shows. We include it because the primary purpose of this metric is to help us figure out if changes are going to result in more questions the community is interested in answering. If we build a feature that helps people find the right site to ask, this metric will help with that too. – Jon Ericson Feb 28 at 20:46

You must log in to answer this question.

Browse other questions tagged .