What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

I'm getting pretty sick of this meme that the title check being run on Stack Overflow (and Super User and Server Fault) is somehow ineffective, obviously broken, etc. etc.

So, I'm going to drop the science hammer on it.

Before we get into the numbers, I'll outline the inspiration and purpose of these title quality checks.

In a nutshell, there are a few phrases ("problem" and "help" being the most obvious) that are really really strong indicators that a post could use some work; so we slap a "no you don't" hurdle in front of users forcing them to rework the titles at least. Titles are really important, both from a "quality of the site" perspective (who wants a homepage full of Help! posts) and from a "getting your question answered" one (if you're hunting for a question to answer, the more detail in a title the better the odds you'll click on it).

To be clear, the desired outcome of the title quality check is to encourage users to add more detail to (and remove useless, low quality indicative, phrases from) their question titles.


I'm looking at the 200 most recently rejected titles that were still eventually posted as questions. Each of these events will be sorted into one of three buckets based on the changes made to the rejected title: actively helpful, benign, and actively harmful.

actively helpful changes are those that result in more detail being added to a question title. This could just be additional technologies being mentioned, or a more thorough re-write (provided it's still the same question, naturally). Examples would be "heap size problem" -> "Android maximum heap size", or "Technology X Question" -> "Technology X Validation".

benign changes are simple deletions of the offending phrase (problem, help, and what have you) or simple substitutions. "Problem with X" -> "Issue with X" or "Heap allocation problem" -> "Can't allocate on heap" would be examples.

actively harmful changes are the "Pr0blems" (or random "s"s, or awkward non-sense that's clearly meant to dodge the filter) that some are insisting happen every time a user trips over this restriction. Also included are "blind idiot deletions", where a user just deletes words until they stumble around the filter, as that's clearly not "trying to make the question better" behavior.

Final outcome?

actively helpful - 37% of all titles gained some detail
benign - 62% of all titles became no worse
actively harmful - 2% (3 total) posts obviously hacked around the filter

We can break down benign even further, into deleted and replaced. Deleted would be posts that removed the offending word (Problem, help, whatever), replaced meaning that something else we don't check for is slapped in there (Issue, confusion, so long winded way of saying Problem, etc.).

Deleting one of these phrases is a net win. They are the salutations of titles, adding nothing. It's not a strong enough win for me to lump them into actively helpful, but it's still a better outcome than posting the original title.

Replacing one of the phrases is a completely wash, GIGO-style. All the filter accomplished in this case is wasting a few moments of someones time (though they arguably deserved it).

Sub outcome?

deleting - 66% of benign modifications removed pointless verbiage
replacing - 33% of benign modifications were just substitutions (though not harmful ones)

Some other quick stats, these 200 posts occurred over a 24 hour and 30 minute period during which 5166 questions were asked (so approximately 4% of all questions tripped this check). Of these 200, there was a single "false positive" (the title really did need "question" in it, in this case); however that post was ultimately deemed off topic and the owner self-deleted it.

Edit

Factoring in reputation, I looked at users > 150 rep. The same ratio holds (76% improved in some way overall; 40% actively helpful, 0% harmful), though our sample size is getting awfully small (43 users). For >2k rep users only 4 posts triggered this, so it's hardly conclusive, but 75% of those were actively helped with the last post being a deletion (so 100% improved in some small way). The theory that reputation indicates title quality is lacking in support, although aforementioned sample size concerns (exacerbated by the rarity of high rep users asking questions) do exist.


tl;dr - the question title check is pretty effective, contrary to what has been asserted (rather blindly) elsewhere. ~78% of all post titles that trip it are improved in at least some small way, 37% in an unambiguous way. "Hacking around" the filter is in fact very very rare.

share|improve this question
2  
Can you share your data, Kevin? –  Michael Petrotta Nov 21 '11 at 23:39
4  
@Michael - there's no way to anonymize this data (since we're looking at specific titles and titles are now unique ), so no I won't be sharing it. I'm not comfortable publicly shaming 200-ish users. –  Kevin Montrose Nov 21 '11 at 23:52
8  
I get that, Kevin, but without it, this hardly seems like either "science" or a good way to close discussion on this question. I think you're begging the question. –  Michael Petrotta Nov 21 '11 at 23:59
1  
(I wouldn't consider it "shaming" particularly, myself.) –  Michael Petrotta Nov 22 '11 at 0:01
10  
"Shaming users"? I don't particularly see the problem. As easy as that... –  slhck Nov 22 '11 at 0:02
    
@slhck: nice, but not the same dataset as what Kevin used. His is only available to devs. –  Michael Petrotta Nov 22 '11 at 0:06
    
@Michael I know, I just wanted to point out that it's not like you can't easily go ahead and query for posts that will somehow shame a lot of users. –  slhck Nov 22 '11 at 0:08
8  
@slhck imgur.com/n5KKj –  Jarrod Dixon Nov 22 '11 at 0:12
8  
Not A Real Question. –  Won't Nov 22 '11 at 16:04
2  
Wow, I'm surprised how effective that is: I am thus no longer opposed to the word filter. It'd be nice if there were some way around it for the 0.5% that need it, but I understand that not being a high priority. –  derobert Nov 26 '11 at 11:02
    
I agree - the filter definitely helps the quality of question titles and the only place it's ever been a problem for me is on StackApps where question titles also double as 'one-liners' promoting an app / library. –  uɐɯsO uɐɥʇɐN Nov 28 '11 at 19:34
9  
-1 didn't notice how that science hammer dropped supports the idea that a veteran higher-reputation user can't use the word "problem" or "question" –  gnat Dec 2 '11 at 17:00
4  
Thanks for the data. But would this not have been better posted as an answer on the current question related to the filter, or as a blog post? –  razlebe Dec 5 '11 at 9:42
3  
Quoting an error message in the title should be allowed. If not the quality of the title and the chance of the question being answered goes down. –  Coyote Jan 3 '13 at 23:18
1  
-1 this is not science. –  Andrew Grimm Aug 27 '13 at 2:31

4 Answers 4

I'm neither upvoting nor downvoting. I have never doubted there is a lot of sub-optimal content that gets improved at least in a small way this way, and I've always admitted as much.

However, the idea that a veteran higher-reputation user can't use the word "problem" or "question" in a post any more - even when it makes perfect sense, or even is the only right term (like in a defined math or programming problem) - remains an insult to our intelligence. (A slight one, but still.) If you have managed to ask a number of good, highly upvoted questions, you are supposed to be trusted by the community to contribute half-way decent content. There is a ton of other things that you get the more trusted you become. Why not a relaxation on these checks as well?

Blocking "problem" and "question" is not a quality filter. It is a filter that uses circumstantial evidence to assert a quality problem with the title. Hence I don't think your argument in the related feature request:

the same quality filters and rules apply to everyone.

really applies.

share|improve this answer
1  
I was going to post a an answer conceding to Kevin's findings, but adding a comment that would turn out just like your answer. So I'm going to +1 this answer accordingly. –  NullUserException อ_อ Nov 22 '11 at 0:26
3  
+1 another fine post by Pekka, I beg the question again, exactly when will Pekka be hired? –  JonH Dec 2 '11 at 20:02
    
@Pekka, in light of Kevin's numbers I think you and I do agree on the moral principle. I wouldn't quite call it an insult, but that's indeed an inconvenience that has bitten me in the past. However, I'm still dubious about how we should implement the thing. You're referring to veteran high-rep users, and that lets me wondering. –  Frédéric Hamidi Dec 9 '11 at 20:57
    
For instance, I've been participating in SO for little more than a year, so I'm hardly a veteran. Should the problem filter apply to me? On the other hand, I have more than 10K rep, should I be immune to it? What about meta, where I have neither seniority nor rep? Should my veteranism on SO allow me to bypass the filter there? All in all, that still looks to me like a very subjective decision to make. Again, I don't know if rep elitism would actually serve us here. –  Frédéric Hamidi Dec 9 '11 at 20:57
2  
@Frédéric if you have 10k rep on any SE site, you have shown that you understand the system to some extent. I think rep elitism would help here. (However, shlck has posted a very interesting suggestion that I find even better: Block, but automatically flag "Problem" attempts in the title as "low-quality" –  Pëkka Dec 9 '11 at 21:02
    
@Pekka, so a 10k rep filter override would apply to all the other sites and their metas? (10k being an example, that might be less.) That's... interesting. Also, shouldn't shlck's post be titled Don't block, but automatically flag “Problem” attempts in the title as “low-quality”? I'm confused (but supportive of the request if I understood it correctly). –  Frédéric Hamidi Dec 9 '11 at 21:12
    
@FrédéricHamidi re the 10k: Yes, that's what I would support. Re the title, you're right!! I'll add a comment. –  Pëkka Dec 9 '11 at 21:16

"Science hammer" huh? Can I get one of those for work? It would be really helpful.

Of course, it's nothing like science, as it's not peer reviewed and there's not enough data for us to evaluate your procedure and results. (As a hint, your rounded percentages don't add up to 100%. If you had provided more raw data, we could calculate the true percentages ourselves.) Plus you've not provided even one poorly-formatted graph. ;-)


But let's concede your analysis: the question filter is super-effective. Now that we have the data, we can begin the debate.

Did you know that wearing a seat belt is super-effective too and that lots of governments have enacted seat-belt laws? But not all of them, because despite saving thousands of lives, seat-belt laws are unpopular. Of course there are some people who don't like seat belts, but even people who do wear them sometimes oppose a law requiring them. Principles such as liberty, personal property and the role of government come into play.

Then there are implementation details. Should Grandpa's Model-T with a top speed of 35 MPH downwind have seat belts retrofitted? In some places, "yes" and in others, "no". Do you need a three-point harness in the front seat or is a simple belt enough? The run-up to the date these laws are enacted is filled with news stories and clarifications about what is and what is not required. People die in car accidents who would have been saved by seat belts in that time period.

There's a (possibly apocryphal) story that when Italy enacted a seat-belt law, it became fashionable for men in Naples to wear white T-shirts with a diagonal line across the chest. To passing Polizie it would look like these men were wearing their seat belts. Do you think showing them statistics that a seat belt will save their life would make them obey the law? Of course not.


You see, there's a subtle difference between changing a line or two of code that affects the users in an unpopular way and changing a legal code. Or rather, it's totally different. And when you move from the technical sphere to the social sphere, the rules of engagement change. It isn't just about truth and numbers and science, it's about how we all are to get along.

I'm all about doing what's right even if it's unpopular. Let's filter the hell out of question titles! Stand firm in your beliefs. Don't back down. But you still gotta treat people well and at least give them the illusion that they have a voice. You gotta at least understand why something is unpopular and wave your hands in the direction of acknowledging them. You gots to stop actin like folks are idiot even when we are.


Look, I'm never going to be affected by this law one way or another and it's none of my business how StackOverflow is governed. But I really hope that some of the smaller StackExchanges I participate on will have the sort of problems that StackOverflow has had almost from the beginning. And if they grow, how can I trust that I and other members of the community will have a say in new laws? Because right now, I see people who are more involved in StackOverflow than I ever was acting seriously pissed off about this law. And it sure sounds like the people in control of that site are not listening to them. And that scares me.

share|improve this answer
1  
So you're saying seatbelts shouldn't be the law because they're (supposedly) unpopular? Also, codinghorror.com/blog/2005/05/… is an interesting read. –  balpha Dec 9 '11 at 18:31
1  
I apologize if I pissed you off, I didn't mean to. I'm honestly trying to understand your metaphor. From all I've seen from you, I had the feeling you're opposing the title filters, but the seatbelt story to me suggests the opposite, which is why I asked how you meant it. Sorry about the misunderstanding, I'll stop bothering you. –  balpha Dec 9 '11 at 18:52
7  
+1 TL;DR version: how can I trust that I and other members of the community will have a say in new laws? Because right now, I see people who are more involved in StackOverflow than I ever was acting seriously pissed off about this law. And it sure sounds like the people in control of that site are not listening to them. And that scares me. –  gnat Dec 9 '11 at 19:13
4  
@gnat if you aren't pissing someone off some of the time, you're not doing anything very useful or interesting. –  Jeff Atwood Dec 9 '11 at 22:24
1  
@Jeff: That sounds like a post hoc ergo propter hoc justification to me. Can you please point me to the place where all of this was discussed beforehand? The closest I could find was this question on "bad titles", but there's not much discussion there. Why ought I trust you? –  Jon Ericson Dec 9 '11 at 22:45
7  
@Jeff - of course the inverse of your logical proposition—if you are pissing people off, you're doing something interesting or useful—isn't necessarily true. In this case it certainly isn't. –  Adam Rackis Dec 10 '11 at 3:30
2  
@adam the data doesn't support your position. –  Jeff Atwood Dec 10 '11 at 4:30
1  
@Jeff - I know you hate pluralization bugs, but it should be the data don't support your position :) But whether this is useful is a value judgement. You're forcing some probably mediocre questions to have slightly better titles, while prohibiting users from asking intelligent ones: halting problem select n + 1 problem NP Complete Problem et cetera, ad infinitum –  Adam Rackis Dec 10 '11 at 4:36
2  
@Jeff, more importantly, the people I've interacted with on SO are some of the smartest people I've ever "met"; it's hard to believe I've learned so much here. If these people are lining in such numbers to decry this policy, I urge you to exercise some epistemic humility and consider the possibility that you might be wrong –  Adam Rackis Dec 10 '11 at 4:43
2  
@adam I urge you to look at the data. –  Jeff Atwood Dec 10 '11 at 4:56
3  
@Jeff, I've looked at the data. I urge you to consider the possibility that changing 50 mediocre questions here and there to have slightly better titles does not justify limiting what good users can ask on this site. Look at the people who disagree with you. –  Adam Rackis Dec 10 '11 at 5:06
2  
@adam it's logic. youtube.com/watch?v=Xa6c3OTr6yA also, this "your theoretical restriction of my rights that I will never actually encounter in practice, is absolutely intolerable!" thing gets really old. –  Jeff Atwood Dec 10 '11 at 5:13
5  
@Jeff - I never claimed it was my right to post questions with the word problem in it; that's a straw man you made up. The fact that you are choosing to put yourself on the opposing side of many of your most valuable contributors to this site betokens a disappointing degree of epistemic arrogance on your part. At some point a truly rational actor would consider the possibility that maybe some of his underlying assumptions used to analyze the data might be wrong. –  Adam Rackis Dec 10 '11 at 5:31
3  
@adam The data tells me this is a non-issue. The internet is a big place; if things are not to your liking here, perhaps some of the millions of other websites might be? Life is too short to be unhappy. –  Jeff Atwood Dec 10 '11 at 5:38
5  
@JeffAtwood if you mean that problem filter is something useful and interesting and that it pissing off someone serves as an indication of that - I agree. I wonder though if you are ready to apply similar reasoning to self? I mean, if say you find out yourself pissed off by guys suggesting to disable the filter for mods / high rep users, would you be ready to consider idea that these guys are doing something useful or interesting? –  gnat Dec 10 '11 at 7:38

Saying that the title filter is a good thing because it usually results in some marginal improvement of a likely marginal question, notwithstanding the occasional halting problem, is like saying that obscenity filters are good because they usually catch naughty words, notwithstanding the occasional sextant, titular, niggardly, spic and span, etc.

share|improve this answer
4  
+1 Thank you. The goal has perhaps been lost. We want better questions, making the titles "better" doesn't make a bad question better. –  MPelletier Dec 10 '11 at 17:11
6  
@problAm - of course, and more importantly, ok questions with bad titles can be edited –  Adam Rackis Dec 10 '11 at 17:24

Perhaps there's a genuinely strong case to be made in favor of the filter - it's hard to judge without access to the raw data - but I'm entirely unimpressed by the statistics given here, for three reasons:

  1. You're not providing us with the raw data to look at, and the classification of changes into the 'helpful', 'benign' and 'harmful' categories is sufficiently subjective that I question whether others would agree with your calls.
  2. Your definition of 'helpful' changes seems to take for granted that any substantial rewrite of a title including the word 'problem' is a helpful one, when it's pretty obvious that a substantial rewrite can be helpful, harmful, benign, or a mixture of helpful and harmful.
  3. Most importantly, your definition of 'harmful' doesn't include the single most important harm that everyone posting about this issue cared about: users not being able to use the word 'problem' in situations where it's the only appropriate word, and a synonym will not do. You have not told us how many of those 200 titles included error messages that had to be moved into the question body (or worse, culled completely by a confused new user), nor how many recognizable formal names of problems had to be twisted into an incorrect or substantially less common version, like the slightly embarrassing 'n+1 selects issue'. Saying that your data shows that the good outweighs the harm when you've only attempted to measure trivial harm that nobody cares about is simply begging the question.
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .