The Efficacy Of Stack Overflow's Question Title Filter

Question

I'm getting pretty sick of this meme that the title check being run on Stack Overflow (and Super User and Server Fault) is somehow ineffective, obviously broken, etc. etc.

So, I'm going to drop the science hammer on it.

Before we get into the numbers, I'll outline the inspiration and purpose of these title quality checks.

In a nutshell, there are a few phrases ("problem" and "help" being the most obvious) that are really really strong indicators that a post could use some work; so we slap a "no you don't" hurdle in front of users forcing them to rework the titles at least. Titles are really important, both from a "quality of the site" perspective (who wants a homepage full of Help! posts) and from a "getting your question answered" one (if you're hunting for a question to answer, the more detail in a title the better the odds you'll click on it).

To be clear, the desired outcome of the title quality check is to encourage users to add more detail to (and remove useless, low quality indicative, phrases from) their question titles.

---

I'm looking at the 200 most recently rejected titles that were still eventually posted as questions. Each of these events will be sorted into one of three buckets based on the changes made to the rejected title: actively helpful, benign, and actively harmful.

actively helpful changes are those that result in more detail being added to a question title. This could just be additional technologies being mentioned, or a more thorough re-write (provided it's still the same question, naturally). Examples would be "heap size problem" -> "Android maximum heap size", or "Technology X Question" -> "Technology X Validation".

benign changes are simple deletions of the offending phrase (problem, help, and what have you) or simple substitutions. "Problem with X" -> "Issue with X" or "Heap allocation problem" -> "Can't allocate on heap" would be examples.

actively harmful changes are the "Pr0blems" (or random "s"s, or awkward non-sense that's clearly meant to dodge the filter) that some are insisting happen every time a user trips over this restriction. Also included are "blind idiot deletions", where a user just deletes words until they stumble around the filter, as that's clearly not "trying to make the question better" behavior.

Final outcome?

actively helpful - 37% of all titles gained some detail
benign - 62% of all titles became no worse
actively harmful - 2% (3 total) posts obviously hacked around the filter

We can break down benign even further, into deleted and replaced. Deleted would be posts that removed the offending word (Problem, help, whatever), replaced meaning that something else we don't check for is slapped in there (Issue, confusion, so long winded way of saying Problem, etc.).

Deleting one of these phrases is a net win. They are the salutations of titles, adding nothing. It's not a strong enough win for me to lump them into actively helpful, but it's still a better outcome than posting the original title.

Replacing one of the phrases is a completely wash, GIGO-style. All the filter accomplished in this case is wasting a few moments of someones time (though they arguably deserved it).

Sub outcome?

deleting - 66% of benign modifications removed pointless verbiage
replacing - 33% of benign modifications were just substitutions (though not harmful ones)

Some other quick stats, these 200 posts occurred over a 24 hour and 30 minute period during which 5166 questions were asked (so approximately 4% of all questions tripped this check). Of these 200, there was a single "false positive" (the title really did need "question" in it, in this case); however that post was ultimately deemed off topic and the owner self-deleted it.

Edit

Factoring in reputation, I looked at users > 150 rep. The same ratio holds (76% improved in some way overall; 40% actively helpful, 0% harmful), though our sample size is getting awfully small (43 users). For >2k rep users only 4 posts triggered this, so it's hardly conclusive, but 75% of those were actively helped with the last post being a deletion (so 100% improved in some small way). The theory that reputation indicates title quality is lacking in support, although aforementioned sample size concerns (exacerbated by the rarity of high rep users asking questions) do exist.

---

tl;dr - the question title check is pretty effective, contrary to what has been asserted (rather blindly) elsewhere. ~78% of all post titles that trip it are improved in at least some small way, 37% in an unambiguous way. "Hacking around" the filter is in fact very very rare.

Answer 1

I'm neither upvoting nor downvoting. I have never doubted there is a lot of sub-optimal content that gets improved at least in a small way this way, and I've always admitted as much.

However, the idea that a veteran higher-reputation user can't use the word "problem" or "question" in a post any more - even when it makes perfect sense, or even is the only right term (like in a defined math or programming problem) - remains an insult to our intelligence. (A slight one, but still.) If you have managed to ask a number of good, highly upvoted questions, you are supposed to be trusted by the community to contribute half-way decent content. There is a ton of other things that you get the more trusted you become. Why not a relaxation on these checks as well?

Blocking "problem" and "question" is not a quality filter. It is a filter that uses circumstantial evidence to assert a quality problem with the title. Hence I don't think your argument in the related feature request:

the same quality filters and rules apply to everyone.

really applies.

Answer 2

"Science hammer" huh? Can I get one of those for work? It would be really helpful.

Of course, it's nothing like science, as it's not peer reviewed and there's not enough data for us to evaluate your procedure and results. (As a hint, your rounded percentages don't add up to 100%. If you had provided more raw data, we could calculate the true percentages ourselves.) Plus you've not provided even one poorly-formatted graph. ;-)

---

But let's concede your analysis: the question filter is super-effective. Now that we have the data, we can begin the debate.

Did you know that wearing a seat belt is super-effective too and that lots of governments have enacted seat-belt laws? But not all of them, because despite saving thousands of lives, seat-belt laws are unpopular. Of course there are some people who don't like seat belts, but even people who do wear them sometimes oppose a law requiring them. Principles such as liberty, personal property and the role of government come into play.

Then there are implementation details. Should Grandpa's Model-T with a top speed of 35 MPH downwind have seat belts retrofitted? In some places, "yes" and in others, "no". Do you need a three-point harness in the front seat or is a simple belt enough? The run-up to the date these laws are enacted is filled with news stories and clarifications about what is and what is not required. People die in car accidents who would have been saved by seat belts in that time period.

There's a (possibly apocryphal) story that when Italy enacted a seat-belt law, it became fashionable for men in Naples to wear white T-shirts with a diagonal line across the chest. To passing Polizie it would look like these men were wearing their seat belts. Do you think showing them statistics that a seat belt will save their life would make them obey the law? Of course not.

---

You see, there's a subtle difference between changing a line or two of code that affects the users in an unpopular way and changing a legal code. Or rather, it's totally different. And when you move from the technical sphere to the social sphere, the rules of engagement change. It isn't just about truth and numbers and science, it's about how we all are to get along.

I'm all about doing what's right even if it's unpopular. Let's filter the hell out of question titles! Stand firm in your beliefs. Don't back down. But you still gotta treat people well and at least give them the illusion that they have a voice. You gotta at least understand why something is unpopular and wave your hands in the direction of acknowledging them. You gots to stop actin like folks are idiot even when we are.

---

Look, I'm never going to be affected by this law one way or another and it's none of my business how StackOverflow is governed. But I really hope that some of the smaller StackExchanges I participate on will have the sort of problems that StackOverflow has had almost from the beginning. And if they grow, how can I trust that I and other members of the community will have a say in new laws? Because right now, I see people who are more involved in StackOverflow than I ever was acting seriously pissed off about this law. And it sure sounds like the people in control of that site are not listening to them. And that scares me.

Answer 3

Saying that the title filter is a good thing because it usually results in some marginal improvement of a likely marginal question, notwithstanding the occasional halting problem, is like saying that obscenity filters are good because they usually catch naughty words, notwithstanding the occasional sextant, titular, niggardly, spic and span, etc.

Answer 4

Perhaps there's a genuinely strong case to be made in favor of the filter - it's hard to judge without access to the raw data - but I'm entirely unimpressed by the statistics given here, for three reasons:

1. You're not providing us with the raw data to look at, and the classification of changes into the 'helpful', 'benign' and 'harmful' categories is sufficiently subjective that I question whether others would agree with your calls.
2. Your definition of 'helpful' changes seems to take for granted that any substantial rewrite of a title including the word 'problem' is a helpful one, when it's pretty obvious that a substantial rewrite can be helpful, harmful, benign, or a mixture of helpful and harmful.
3. Most importantly, your definition of 'harmful' doesn't include the single most important harm that everyone posting about this issue cared about: users not being able to use the word 'problem' in situations where it's the only appropriate word, and a synonym will not do. You have not told us how many of those 200 titles included error messages that had to be moved into the question body (or worse, culled completely by a confused new user), nor how many recognizable formal names of problems had to be twisted into an incorrect or substantially less common version, like the slightly embarrassing 'n+1 selects issue'. Saying that your data shows that the good outweighs the harm when you've only attempted to measure trivial harm that nobody cares about is simply begging the question.