Heuristics for detecting a bad answer?

Question

A lot of bad answers are slipping through the cracks on Stack Overflow. You can see some examples at

Thanks a lot for this post ... and other first time user curiosities

I've been paging through hundreds of these first answers by new users and I've identified a few heuristics that, when applied at the time of answer submit for new users, could drastically reduce the amount of noise answers in the system.

https://stackoverflow.com/review

(route still experimental for now, but very likely to be deployed with some future improvements)

First of all, we allowed 'answers' with a minimum length of 15 characters. I blame myself for this terrible choice, since very short answers are almost always crap. For some crazy reason I used the same minimum constant that we use for comments! No longer. Effective immediately the new minimum answer size is 30 characters. I may increase it a bit more, even, but doubling it seems safe-ish for now.

Beyond the length test -- which is HUGELY important -- there are a bunch of other factors we can use to calculate a "bad answer weight", again, based on me personally browsing through hundreds of bad answers by new users and identifying patterns I observed:

Word checks

• contains word "help"
• contains word "test" or "testing"
• contains word "thanks", "thx", "cheers", "great"
• contains word "bump"
• contains word "same", "problem"
• contains word "sorry"
• contains word "work", "working"
• contains ":)" or ":-)" or ":(" or ":-("

Character set checks

• contains one or more exclamation points
• contains one or more question marks
• contains all (or mostly) uppercase characters
• contains all lowercase characters
• does not contain any spaces
• does not contain any ascii chars
• contains .., possibly repeated

Content checks

• includes hyperlink type text (even without the http://)
• includes email address
• content ends in question mark
• answers with low entropy, eg, repeated characters like "asdfasdfasdfjkjkjk"

In my experience having clicked through hundreds of bad answers, these heuristics, particularly when combined, would get nearly all of them. So then, the goal: to prevent these low-quality answers from entering the system at all!

These would all be rolled up into an aggregate answer score, which represents a confidence threshold for every new answer by a new user.

If the score is very low, then the answer will simply be rejected outright. If the score is below a (configurable) threshold, we will present the user with an answer EULA and some basic help, like:

This answer doesn't appear to meet our quality thresholds. Do you still want to submit it to {sitename}? [ ] Yes [ ] No

Clicking yes on the EULA will automatically flag the answer for moderator attention.

Edit: this heuristic is now live (sans the specific word checks, although smileys are penalized) at

https://stackoverflow.com/review/low-quality-posts

Please try it for yourself!

Answer 1

Collect a database of few hundred good and bad answers and make some machine learning -- good rules will appear automatically. Even better, you can make a challenge out of it -- post a 2/3 of this base with answers and 1/3 without, tell people to predict the missing answers, obtain their results, pick best, give the winner a unicorn-branded mug and deploy the algorithm she/he used. (Small scale version of what Netflix did)

Answer 2

contains one or more exclamation points

I've written answers with exclamation marks. They're more than appropriate where they appropriate.

content ends in question mark

I've written such answers. Not because I was making a question out of my answer but to put my answer in a certain literary style.

- contains word "help"
- contains word "test" or "testing"
- contains word "same", "problem"
- contains word "sorry"
- contains word "work", "working"

A huge number of good answers would contain those! You wouldn't believe how many!

Then again, you know better right? Right?

Answer 3

Any answer that includes text-speak:

• teh
• hai
• l8r
• pls
• thx
• dis
• wats

etcetera

Answer 4

As mbq has hinted at with the rather vague term "machine learning", the way to do this is not to try to invent your own heuristics using a blacklist; you of all people know the dangers of blacklists, Jeff.

Instead what you want to do is collect as many posts as you can fitting the "not an answer" or "not even good enough to downvote" criteria and run a Bayesian analyzer on them. You could do the same for actual spam posts, but don't do them together. Also perhaps implement the proposed Flag as "Not An Answer'", at least for moderators if not for 20k, which would allow the list to be maintained.

This is adaptive, self-maintaining, would work on every SE (as long as you disable it until the corpus is large enough to give a high confidence interval), and a good deal more reliable than re-inventing the wheel here.

That said, there are a few heuristic checks you could use that go beyond simple word/character analysis. The e-mail address check is a good one. Here are a few more:

• Posts whose content is more than X% hyperlinks (suggested X ~ 30). Usually these take the form of have a look at [this page](http://example.com). These might be spam, might not; it's usually impossible to tell the difference without following the link. At best, they are usually useless answers that provide no original content nor context for the link. This is similar to your proposed hyperlink check, but far too many legitimate answers contain hyperlinks for a naïve check to be a useful metric; in fact, hyperlinks are often the sign of a good answer on account of citations, but those answers will only have a small percentage of hyperlinks.

• Short answers to hot/popular questions or questions that already have many answers. These are usually indicative of people trying to jump on the bandwagon and get some cheap rep/badges by rattling off mindless quips. You've also probably noticed that after 20 or 30 answers you start to get mass duplication and a very low SNR. The key word here is "short", because a long answer might indicate that the author thought the other answers were garbage and wanted to set the record straight.

• Short answers, period. You've already stated that you've increased the minimum, but that doesn't mean you can't use the length as an additional heuristic. If an answer is short enough to fit in a comment (150 characters), there's a reasonably good probability that it should have been a comment. Biasing the system against very short answers also makes it more likely to catch answers where somebody adds crap like (30 characters!) to the end of their answer to make it meet the hard limit.

• Answers submitted in less than X seconds (with X ~ 20 maybe) from the first character being typed. Again, this tends to be a sign of very little thought and zero proofreading going into an answer. Obviously this could be circumvented rather easily, but the kind of garbage we're worried about tends to come from people who either wouldn't know or wouldn't care enough to figure out how.

• Reputation as an inverse heuristic. One has to assume that if a member has enough reputation to edit other people's posts then the system trusts them to create their own, however many other heuristic checks fail. A sliding scale starting at 100 rep and going up to "edit" rep (2000) would probably work.

Of course, these are all weak heuristics compared to a proper Bayesian filter and some might work better than others. Nevertheless, it wouldn't hurt to try them out and see which ones increase the hit rate or decrease the false positive rate.

Answer 5

I would be careful using word based heuristics.

For example:

Based on your problem, the following solution should work for you:

some code that proves p = np

I had the same problem a few weeks ago, and that's how I solved it!

The following link should help you understand your problem better:

http://en.wikipedia.org/wiki/P_versus_NP_problem

Answer 6

Answers (and questions) containing email addresses are suspicious too.

There is no need to provide your email, and it is easy to filter these out.

Another option is to search for swearwords. I have seen them, but never in good answers. But of course we shouldn't make the clbutic mistake.

And don'nt forget the word noob with possibly a repetition of n, o and b and sometimes ending with s. (like noooooooooobbbbb).

Answer 7

I'd add "overuse of the dot" into "character checks"..........using the SuperMegaEllipsis is a pretty strong red flag: \.{4,}

Use of the ellipsis is a ... lesser flag, as it could have valid uses. Double dot..is usually a sign of bad grammar - more serious than an ellipsis, but less than "I buy my dots at a bulk rate" IMHO.

Answer 8

Increasing the minimum length will not help much, as bad answers will just be made longer. Using it as a filter to show high rep users answers that may be bad, that could then be voted on to delete, may work very well.

I am now thinking 3 systems need to be in place.

• A warning that is hard to ignore if a new user posts an answer that the filter thinks is bad.
• A way for established users to vote to delete bad answers, the number of votes needed should be a lot less if the answer is from a new users and gets a bad number on the filter.
• A way that high rep users can get a list of “likely bad answers”.

Down voting bad answers doesn’t work, as they don’t get deleted and very often someone else will up vote them as it is not considered nice to down vote an answer from a new user. However using a filter like “HasDownVotes, and IsNewUser” may be a good way to get a list of answers that high rep users then vote on to delete.

Answer 9

Having just tried review/low-quality-posts on gaming I must say it's already very nice and helped me find some problematic posts, but I would recommend removing highly-upvoted answers from there. If something has 8 upvotes and no downvotes, no matter how poorly it ranks, it's probably not of low quality.

Answer 10

thx a bunch so relly needed this!!! cheers

Answer 11

I think you could add non-capitalized 'i's to the metric for determining post quality.

OK, so you may think I'm a racist or a grammar nazi, but there is a strong correlation with question quality...

@Pekka pointed out that there might be false positives with i's in code. You could always start off with posts containing NO capital letters at all... Scrap that, @Jeff says you do that already.

Answer 12

Clicking yes on the EULA will automatically flag the answer for moderator attention.

This sounds good in theory, but there's a big potential for the mod queue to explode with these flags.

Anyone who is writing a low-quality answer is going to ignore this. Completely. Except possibly if you exploit the UI to make it such that clicking the "expected" button actually cancels the submit... but that's not a very friendly thing to do.

You could implement a subsequent system of rate-limiting, but if you get a lot of unique views, then that will have a limited effect. Not to mention that such a system is complex.

Honestly I don't know what to suggest as an alternative besides setting the hard threshold fairly high.

---

I also have an heuristic suggestion: answer time relative to question asked time or last modified time. I see a lot of "thank you answers" added much much later than the main activity in a question. Even just that combined with a <100 character criteria is pretty much always a low-quality answer.

Answer 13

I also find it most irritating when some newcomer puts a footnote to his every answer:

Best regards,
Dork

or

Please get in touch with me,
Dork

or

If you need help let me know,
Dork

This is self-promotion in the most obtrusive way. Should be discouraged.

Answer 14

There might be one class of answer that needs to be allowed - though not as an answer.

This answer has just been posted on a 5 month old question. It basically says:

I'm getting this problem as well, did you ever find a solution.

Now at first glance this looks like a bad answer, but there is no other answer on the question - not even from the OP - so either they solved their problem and didn't bother to come back to update the question or abandoned the approach altogether. Either way an answer from the OP (or deletion in the latter case) would be useful.

If the problem was solved we'd have an answer, if not we'd know that this approach doesn't work and should try something else. If the question was deleted the new poster would be free to post their own question!

So in this case the answer should act as a reminder to the OP to "close" the question in some way.

Now I'd be the first to point out that posting this as an answer is wrong and not what SO was designed for, so do we need some other way for low rep users to be able to interact with a question to either bump it to the "interesting" page or prod the OP so that it gets some more attention?

I don't know what that interaction might be, but reminding the OP about these questions can't be a bad thing.

Answer 15

Character set checks

• contains one or more exclamation points
• contains one or more question marks
• contains all (or mostly) uppercase characters
• contains all lowercase characters
• does not contain any spaces

• does not contain any ascii chars

  Content checks

• includes hyperlink type text (even without the http://)

• includes email address
• content ends in question mark
• answers with low entropy, eg, repeated characters like "asdfasdfasdfjkjkjk"

This sounds like an editor's dream, rather than a spam detection system :)

When I look at MSO's low quality posts, I see hardly any spam (well, there are some posts of mine trying to see the quality systen at work) -- but I see a lot of posts that could do with some basic copyediting (1 2 3 4 5).

It is particularly satisfying to see one such bad answer, editing it and watching its score increase. Maybe you could use it as a metric to judge the quality of edits -- did the score increase?

In the same vein I would like to add:

• contains four or more periods ("that was great.....thanks man.....")
• contains unformatted hyperlinks
• contains "30 char" or other forms of content-less padding
• is a self-answer or a late answer.

Answer 16

(I'm talking about the /review page here)

Definitely worth taking votes into account: this one showed up as low quality, but it's accepted with (currently) 12 upvotes.

Also, could you have some way of marking things as "Taken care of", so other people know they don't have to bother? (Or at least as a marker for myself). If you don't trust one person to 'TCO', you could have a 5-vote style system like for closing...

Answer 17

I'm surprised not to see this simple suggestion in the existing answers:

• short text (like "try this link" or "check this out", but it's not necessary to analyze the text)
• immediately followed by a link

Those kind of short answers:

try it out, this worked for me http://bozoland.com/fix-syntax-error

are generally flagged as they are "link only answers" and hit the NAA/VLQ (I deleted so many of those) so why not preventing such posts? Maybe only for new users who don't know the rules of the site yet?

If the user decides to workaround the block by adding some blurb like

fdfjsdfjsjdfkjsd

this can be detected by heurstics as well, and if it can't, then nevermind. At least those short link only answers won't make us waste our time in the VLQ/NAA queue.

Even if the link points on another answer of the same SE site, it's a sign of bad answer (the question should be flagged as a duplicate instead)

Answer 18

I like the idea of a challenge (hat tip to @mbq) and I think it would be a good way to determine how to identify badness.

What about publishing a subset of the SO database with 200 answers that moderators rate as good, 200 bad and 200 indifferent. Then offering a bounty for the most elegant sql statement that categorizes the answers into the three buckets - for extra kudos don't tell anyone beforehand which is which.

Answer 19

Another phrase that I've seen often in non-answers is "Could you", as in "Bob, could you explain your answer?" This appears in many actual answers ("Could you use XXX for this? It has these advantages and I've used it in the past."), but it seems more common in non-answers.

It might be nice to have as another filter item on the /review page, along with the "thanks" one that's already there.

Answer 20

I'd like to see a link to http://hyperboleandahalf.blogspot.com/2010/04/alot-is-better-than-you-at-everything.html appear when someone uses "alot", but others didn't like this suggestion alot.

Answer 21

Unless I'm misunderstanding, I would say that including hyperlinks is also an indicator for a good answer. I often go back and edit my answers to include hyperlinks to reference material (man pages, MSDN, etc.) or blog posts.

Also, it's not clear from your post, but I'm assuming that the heuristics would not count code. An answer that's just "This should help:" followed by a block of C code could easily have exclamation marks, question marks, :(, and all the letters but one would be lowercase. Of course an answer in COBOL would probably be a huge wall of uppercase text. I'd hate to see all COBOL answers flagged as bad.

Answer 22

Also from our earlier days, you can safely ban every answer containing a name or a link to any web hosting company. This is pure spam.