What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 134 Stack Exchange communities.

A lot of bad answers are slipping through the cracks on Stack Overflow. You can see some examples at

Thanks a lot for this post ... and other first time user curiosities

I've been paging through hundreds of these first answers by new users and I've identified a few heuristics that, when applied at the time of answer submit for new users, could drastically reduce the amount of noise answers in the system.

http://stackoverflow.com/review

(route still experimental for now, but very likely to be deployed with some future improvements)

First of all, we allowed 'answers' with a minimum length of 15 characters. I blame myself for this terrible choice, since very short answers are almost always crap. For some crazy reason I used the same minimum constant that we use for comments! No longer. Effective immediately the new minimum answer size is 30 characters. I may increase it a bit more, even, but doubling it seems safe-ish for now.

Beyond the length test -- which is HUGELY important -- there are a bunch of other factors we can use to calculate a "bad answer weight", again, based on me personally browsing through hundreds of bad answers by new users and identifying patterns I observed:

Word checks

  • contains word "help"
  • contains word "test" or "testing"
  • contains word "thanks", "thx", "cheers", "great"
  • contains word "bump"
  • contains word "same", "problem"
  • contains word "sorry"
  • contains word "work", "working"
  • contains ":)" or ":-)" or ":(" or ":-("

Character set checks

  • contains one or more exclamation points
  • contains one or more question marks
  • contains all (or mostly) uppercase characters
  • contains all lowercase characters
  • does not contain any spaces
  • does not contain any ascii chars
  • contains .., possibly repeated

Content checks

  • includes hyperlink type text (even without the http://)
  • includes email address
  • content ends in question mark
  • answers with low entropy, eg, repeated characters like "asdfasdfasdfjkjkjk"

In my experience having clicked through hundreds of bad answers, these heuristics, particularly when combined, would get nearly all of them. So then, the goal: to prevent these low-quality answers from entering the system at all!

These would all be rolled up into an aggregate answer score, which represents a confidence threshold for every new answer by a new user.

If the score is very low, then the answer will simply be rejected outright. If the score is below a (configurable) threshold, we will present the user with an answer EULA and some basic help, like:

This answer doesn't appear to meet our quality thresholds. Do you still want to submit it to {sitename}? [ ] Yes [ ] No

Clicking yes on the EULA will automatically flag the answer for moderator attention.

Edit: this heuristic is now live (sans the specific word checks, although smileys are penalized) at

http://stackoverflow.com/review/low-quality-posts

Please try it for yourself!

share|improve this question
1  
Would it be hard to implement multiple EULAs? Like the examples in Thanks a lot for this post ... and other first time user curiosities show some specific hints for specific cases. And why only for new users? (If only for "contains all lowercase characters", I'd say: apply to all answers, not just for new users.) –  Arjan Dec 17 '10 at 9:40
    
Intersting, I just checked the 12 answers containing "Thank you". Only two more or less usable answers, 7 comments, 1 other question, and some spam. –  Toon Krijthe Dec 17 '10 at 9:44
    
This one slips through on the "Thanks" filter for the month? stackoverflow.com/questions/4357252/weblogic-managed-server/… –  JoseK Dec 17 '10 at 9:56
    
@Jose - in that case it's the OP whose managed to get 2 accounts. I've flagged the answer and hopefully the users will be merged. –  ChrisF Dec 17 '10 at 10:16
3  
How about a self-adapting filter based on probability of words like most e-mail clients use? I think most of your criteries might geht thrown in there automatically and probably quite a bit more. In my experience such AI filters work better than human made assumptions. –  Georg Schölly Dec 17 '10 at 11:35
4  
so are these individual heuristics weighted? i imagine "does not contain any spaces" should affect the score by much more than "contains one or more question marks", for example –  Kip Dec 17 '10 at 15:03
4  
(EULA is a bit of misnomer isn't it?) –  Andreas Bonini Dec 17 '10 at 17:25
1  
I was about half-way through and thinking "What a great way to filter bad questions!" Oops. –  Uphill Luge Dec 17 '10 at 18:24
    
@hans we already have the question eula blog.stackoverflow.com/2010/10/asking-better-questions –  Jeff Atwood Dec 17 '10 at 18:37
2  
When I see a non-answer answer now, I flag for moderator attention. It sounds like, in many cases, the heuristics above will catch those answers and flag for moderator attention. How can I avoid wasting time? Can "possibly-bad" answers be flagged on, say, the 10k reports? Or maybe I should stop flagging myself, let the heuristics handle it. –  Michael Petrotta Dec 17 '10 at 19:05
14  
I'm amused that we've established so formal a consensus that smileys are a useful marker for questionable content. –  dmckee Dec 17 '10 at 20:38
4  
Don't forget backward smilies. –  Lance Roberts Dec 19 '10 at 6:49
3  
This day will proudly mark the occasion when we made triple damn sure that there are no fun questions! Down with the smileys!!! On a more serious note, lower score is worse than higher score, correct? It seems so from this question, but it's not entirely clear in the review - could we get a "sort by score", please? –  Piskvor Dec 20 '10 at 12:36
1  
You're missing ):, (:, etc. –  Nick T Apr 26 '12 at 17:55
    
@NickT I always use (: in other pages. Don't think I've ever used it in Stack Overflow, but some users might use it. –  Francisco Presencia Sep 29 '12 at 10:00

21 Answers 21

Unless I'm misunderstanding, I would say that including hyperlinks is also an indicator for a good answer. I often go back and edit my answers to include hyperlinks to reference material (man pages, MSDN, etc.) or blog posts.

Also, it's not clear from your post, but I'm assuming that the heuristics would not count code. An answer that's just "This should help:" followed by a block of C code could easily have exclamation marks, question marks, :(, and all the letters but one would be lowercase. Of course an answer in COBOL would probably be a huge wall of uppercase text. I'd hate to see all COBOL answers flagged as bad.

share|improve this answer
    
code blocks are excluded from such calculations. If you fail to enter things in a code block, well.. low quality :) –  Jeff Atwood Mar 21 '11 at 6:04

I think you could add non-capitalized 'i's to the metric for determining post quality.

OK, so you may think I'm a racist or a grammar nazi, but there is a strong correlation with question quality...

@Pekka pointed out that there might be false positives with i's in code. You could always start off with posts containing NO capital letters at all... Scrap that, @Jeff says you do that already.

share|improve this answer
1  
What's racist about "I" vs. "i"? [scratches head in confusion] Apart from that, yes, poor grammar correlates with low quality, and referring to self as "i" is a very common sign of poor grammar here. –  Piskvor Mar 3 '11 at 8:32
    
@Piskvor: Some people have inferred that shift-deficiency is at least partially related to nationality. Any correlation is left as an exercise for the reader. The management declines all responsibility. –  Benjol Mar 3 '11 at 8:35
    
[facepalm] –  Piskvor Mar 3 '11 at 8:44

Another phrase that I've seen often in non-answers is "Could you", as in "Bob, could you explain your answer?" This appears in many actual answers ("Could you use XXX for this? It has these advantages and I've used it in the past."), but it seems more common in non-answers.

It might be nice to have as another filter item on the /review page, along with the "thanks" one that's already there.

share|improve this answer

I like the idea of a challenge (hat tip to @mbq) and I think it would be a good way to determine how to identify badness.

What about publishing a subset of the SO database with 200 answers that moderators rate as good, 200 bad and 200 indifferent. Then offering a bounty for the most elegant sql statement that categorizes the answers into the three buckets - for extra kudos don't tell anyone beforehand which is which.

share|improve this answer

(I'm talking about the /review page here)

Definitely worth taking votes into account: this one showed up as low quality, but it's accepted with (currently) 12 upvotes.

Also, could you have some way of marking things as "Taken care of", so other people know they don't have to bother? (Or at least as a marker for myself). If you don't trust one person to 'TCO', you could have a 5-vote style system like for closing...

share|improve this answer
    
I am in favor of "reviewed by x users" threshold for coloration, but I think X should be high for the time being, 20-30 at least. Till we are fairly sure it's not being abused. –  jcolebrand Dec 21 '10 at 13:16
    
@drachenstern, maybe the review page should be reserved for higher-rep users? –  Benjol Dec 21 '10 at 13:18
    
I think post 1500 rep myself –  jcolebrand Dec 21 '10 at 13:19

Having just tried review/low-quality-posts on gaming I must say it's already very nice and helped me find some problematic posts, but I would recommend removing highly-upvoted answers from there. If something has 8 upvotes and no downvotes, no matter how poorly it ranks, it's probably not of low quality.

share|improve this answer
2  
I'd disagree. I found a 6 vote answer that was just a single word linked to somewhere else. I left a comment and the answerer updated the answer with some relevant information. A win I'd say. –  ChrisF Dec 20 '10 at 15:54
    
We already sort to the bottom partially based on score. I agree with ChrisF we want it on the list, just not at the top. –  Jeff Atwood Dec 20 '10 at 22:15
1  
+1, the best possible heuristic is what do other SO users think. –  j_random_hacker Dec 21 '10 at 2:07

Character set checks

  • contains one or more exclamation points
  • contains one or more question marks
  • contains all (or mostly) uppercase characters
  • contains all lowercase characters
  • does not contain any spaces
  • does not contain any ascii chars

    Content checks

  • includes hyperlink type text (even without the http://)

  • includes email address
  • content ends in question mark
  • answers with low entropy, eg, repeated characters like "asdfasdfasdfjkjkjk"

This sounds like an editor's dream, rather than a spam detection system :)

When I look at MSO's low quality posts, I see hardly any spam (well, there are some posts of mine trying to see the quality systen at work) -- but I see a lot of posts that could do with some basic copyediting (1 2 3 4 5).

It is particularly satisfying to see one such bad answer, editing it and watching its score increase. Maybe you could use it as a metric to judge the quality of edits -- did the score increase?

In the same vein I would like to add:

  • contains four or more periods ("that was great.....thanks man.....")
  • contains unformatted hyperlinks
  • contains "30 char" or other forms of content-less padding
  • is a self-answer or a late answer.
share|improve this answer
    
the .. penalty is already there, actually. The more .. the more it is penalized. Hyperlink, we already test for very generically (no formatting required) so that would get picked up. –  Jeff Atwood Dec 20 '10 at 12:48
    
@Jeff Well, ellipsis aren't necessarily bad... –  badp Dec 20 '10 at 12:49
    
radp, he's saying that .. is penalized, .... is penalized, ...... is penalized, etc. An odd number - such as the 3 dots usually used for ellipsis - won't get penalized. –  Oak Dec 21 '10 at 13:19
    
@Oak ...that'd be odd. –  badp Dec 21 '10 at 15:51

thx a bunch so relly needed this!!! cheers

share|improve this answer
    
yeah, we're not using the words (although there is a smiley penalty) because the existing non-word heuristics were so effective when combined together. –  Jeff Atwood Dec 20 '10 at 11:57
    
@Jeff This answer still topped the charts for MSO today :) –  badp Dec 20 '10 at 12:14
    
-1 for using spaces and other ASCII characters. –  Gabe Mar 21 '11 at 5:32

As mbq has hinted at with the rather vague term "machine learning", the way to do this is not to try to invent your own heuristics using a blacklist; you of all people know the dangers of blacklists, Jeff.

Instead what you want to do is collect as many posts as you can fitting the "not an answer" or "not even good enough to downvote" criteria and run a Bayesian analyzer on them. You could do the same for actual spam posts, but don't do them together. Also perhaps implement the proposed Flag as "Not An Answer'", at least for moderators if not for 20k, which would allow the list to be maintained.

This is adaptive, self-maintaining, would work on every SE (as long as you disable it until the corpus is large enough to give a high confidence interval), and a good deal more reliable than re-inventing the wheel here.

That said, there are a few heuristic checks you could use that go beyond simple word/character analysis. The e-mail address check is a good one. Here are a few more:

  • Posts whose content is more than X% hyperlinks (suggested X ~ 30). Usually these take the form of have a look at [this page](http://example.com). These might be spam, might not; it's usually impossible to tell the difference without following the link. At best, they are usually useless answers that provide no original content nor context for the link. This is similar to your proposed hyperlink check, but far too many legitimate answers contain hyperlinks for a naïve check to be a useful metric; in fact, hyperlinks are often the sign of a good answer on account of citations, but those answers will only have a small percentage of hyperlinks.

  • Short answers to hot/popular questions or questions that already have many answers. These are usually indicative of people trying to jump on the bandwagon and get some cheap rep/badges by rattling off mindless quips. You've also probably noticed that after 20 or 30 answers you start to get mass duplication and a very low SNR. The key word here is "short", because a long answer might indicate that the author thought the other answers were garbage and wanted to set the record straight.

  • Short answers, period. You've already stated that you've increased the minimum, but that doesn't mean you can't use the length as an additional heuristic. If an answer is short enough to fit in a comment (150 characters), there's a reasonably good probability that it should have been a comment. Biasing the system against very short answers also makes it more likely to catch answers where somebody adds crap like (30 characters!) to the end of their answer to make it meet the hard limit.

  • Answers submitted in less than X seconds (with X ~ 20 maybe) from the first character being typed. Again, this tends to be a sign of very little thought and zero proofreading going into an answer. Obviously this could be circumvented rather easily, but the kind of garbage we're worried about tends to come from people who either wouldn't know or wouldn't care enough to figure out how.

  • Reputation as an inverse heuristic. One has to assume that if a member has enough reputation to edit other people's posts then the system trusts them to create their own, however many other heuristic checks fail. A sliding scale starting at 100 rep and going up to "edit" rep (2000) would probably work.

Of course, these are all weak heuristics compared to a proper Bayesian filter and some might work better than others. Nevertheless, it wouldn't hurt to try them out and see which ones increase the hit rate or decrease the false positive rate.

share|improve this answer
    
if Bayesian is so amazingly perfect, why do I end up with spam in multiple gmail accounts every single day, and on multiple blogs? I'm very confident in the above heuristics. –  Jeff Atwood Dec 18 '10 at 20:31
8  
@Jeff: Who said it was amazingly perfect? I said it would be better than reinventing the wheel, that's all. Do you really think that an ad-hoc, seat-of-the-pants approach is going to do be better? I'm confident in some of the heuristics you mentioned too, but the devil's in the details, and most of what you've mentioned is already more than adequately covered by a simple Bayesian algorithm. –  Aarobot Dec 18 '10 at 21:54
2  
I really like the "reputation as inverse" heuristic. I find myself using this in my own thinking when I look down the list of spam flags. A user with 1-100 reputation will cause me to very seriously look at what they're saying, whereas a user with 1000+ reputation is most likely not posting spam (they may have posted one relevant link to their company's product, and someone got a little overzealous with the flagging). –  Brad Larson Dec 19 '10 at 14:56
    
Yes @Brad, although it is important to note that it is just another heuristic check and not an immunity; I've seen members over and above 1000 rep posting utter crap, although rarely at 2000 rep or above. –  Aarobot Dec 19 '10 at 16:53
    
@Aarobot - True, no heuristic will stand by itself (except perhaps answer shortness). We wouldn't want to prevent low-reputation users from posting, because otherwise how will people get started. I guess I envision this being a combination, with if it hits one or more of the above "bad question" criteria and it's from a low-rep user, it's far more likely to trip the filter than another question hitting the criteria from a high-rep user. I've posted answers that had "thanks to xxx", etc. in them, but were real answers. –  Brad Larson Dec 19 '10 at 21:41
1  
Interesting. However, re "Answers submitted in less than X seconds" - I'd make it "Last edit in less than X seconds", as many answers are edited a lot within the grace period. When I'm answering, I usually type out a brief answer, post, and then edit it into a full-sized version, in 3+ iterations. This seems a common way to adapt to SO's "Fastest gun in the West problem". See also: meta.stackexchange.com/questions/9731/… –  Piskvor Dec 20 '10 at 9:16
2  
@Piskvor: It's fine if you do that, but some members post utter crap as their first answer just so they can edit it within the grace period, and that is not helpful or productive. It's gaming the system and those answers should be stopped at the gate. If your 20-second answer contains sufficient detail to actually answer the question then great; otherwise, you should put more effort into it the first time around. –  Aarobot Dec 20 '10 at 17:44
3  
"Answers submitted in less than X seconds " is wrong, I often type my answser is ms-word, to get spell checking etc, then copy it in. So the system thinks I typed very fast.... –  Ian Ringrose Dec 20 '10 at 20:47
2  
@Ian: And if it's otherwise a good and detailed answer (which seems likely if you went to the trouble of writing it in MS Word), then the system wouldn't have a problem with it. Heuristics are not rules. –  Aarobot Dec 20 '10 at 22:04
3  
@Jeff That's probably because the vast majority of messages sent to your Gmail addresses are spam. The filter may be 99.8% sensitive, but there's enough messages left over in that 0.2% to irritate you. –  Patrick McElhaney Dec 21 '10 at 14:14
2  
And machine learning is not a vague term, it is a well defined science; also about much more powerful methods than Bayesian filters. –  mbq Dec 22 '10 at 22:53
    
You are correct, @mbq, there are var more powerful machine-learning methods than Bayesian filtering. What I meant was that you didn't specify what type of machine learning, and the term could refer to several different approaches. I picked the lowest common denominator here. –  Aarobot Dec 23 '10 at 15:35
    
@Aarobot Precisely, I meant that selecting the right approach is the part of the machine learning -- and thus should be a part of such challenge. –  mbq Dec 23 '10 at 15:46
    
@Jeff: Last I checked, Gmail doesn't use Bayesian filtering. –  Gabe Mar 21 '11 at 5:31

Any answer that includes text-speak:

  • teh
  • hai
  • l8r
  • pls
  • thx
  • dis
  • wats

etcetera

share|improve this answer
15  
...but not i18n and l10n. –  Piskvor Dec 18 '10 at 13:51
1  
Instinctively +1ed this but actually I feel a bit guilty now... Text-speak grates on me big-time but is there really a correlation with low question quality? I suspect it also makes things harder for people whose 1st language isn't English. –  j_random_hacker Dec 21 '10 at 1:47
5  
@j_random - I strongly suspect there's a correlation. Wouldn't want to pick on non-native speakers, so if some turn out to be typos more than deliberate they could be left out. Remember though, we're not talking insta-ban for using these strings, just something that's weighted along with other factors. Using one or two of these would probably pass as long as there aren't other warning signs as well. –  Brad Mace Dec 21 '10 at 1:57
    
Some of these (pls springs to mind) are very common among those who learned their English online. It would be useful to cross these with geo-IP for better correlation with english usage patterns in an area, but that's kind of reaching. –  sysadmin1138 Dec 21 '10 at 3:43
4  
@sys Regardless of why people write like this, if they do it a lot they lower the quality of the site. That type of content is a big part of why I'm not worried about Facebook Questions (meta.stackexchange.com/questions/72756) –  Brad Mace Dec 21 '10 at 3:47
4  
we did end up adding txt-speak detection because it turned out this is a very strong signal of low quality. Also added: lower case i, d (the), y (why), u (you), r (are). –  Jeff Atwood May 9 '11 at 8:49

There might be one class of answer that needs to be allowed - though not as an answer.

This answer has just been posted on a 5 month old question. It basically says:

I'm getting this problem as well, did you ever find a solution.

Now at first glance this looks like a bad answer, but there is no other answer on the question - not even from the OP - so either they solved their problem and didn't bother to come back to update the question or abandoned the approach altogether. Either way an answer from the OP (or deletion in the latter case) would be useful.

If the problem was solved we'd have an answer, if not we'd know that this approach doesn't work and should try something else. If the question was deleted the new poster would be free to post their own question!

So in this case the answer should act as a reminder to the OP to "close" the question in some way.

Now I'd be the first to point out that posting this as an answer is wrong and not what SO was designed for, so do we need some other way for low rep users to be able to interact with a question to either bump it to the "interesting" page or prod the OP so that it gets some more attention?

I don't know what that interaction might be, but reminding the OP about these questions can't be a bad thing.

share|improve this answer
2  
I do find that the site lacks a decent interface for 'closing off' abandoned questions or should we simply delete these if there's no useful answer? –  Ivo Flipse Dec 17 '10 at 22:50
6  
I kind of violently disagree with this. If they must post an 'answer' they should be researching the problem and providing partial fixes or some other hints and clues that get everyone toward an eventual solution. A brain damaged "I have this problem too!" is not helping anyone. –  Jeff Atwood Dec 17 '10 at 22:51
1  
@Jeff - I had assumed that they'd done some research which is why they found the question in the first place. I agree that the current "answer" doesn't really help anyone, which is why I was trying to think of a solution. Perhaps if you can detect the "me too" nature of the answer it's still blocked as described but a message sent to the OP? –  ChrisF Dec 17 '10 at 22:54
    
The I have this problem too answer seems to be all too prevalent on the Stack Exchange sites...Is it possible to somehow limit these or clean them up? Is flagging appropriate for this? –  studiohack Dec 18 '10 at 18:23
    
@studiohack - yes flagging is appropriate –  ChrisF Dec 18 '10 at 18:24
    
I think ideally upvoting a question would be the way to indicate "I'm getting this too", so I would suggest that upvoting a question marks it as "bumped" in the same way as commenting/editing/answering it (maybe it already is?) –  j_random_hacker Dec 21 '10 at 2:05
3  
@j_radom_hacker - you need 15 rep to vote up, so it's not an option available to completely new users. –  ChrisF Dec 21 '10 at 8:43
    
I don't get why an answer is necessary here, except on very new users trying to figure out the system. Otherwise I would say "it should be a comment" since that alerts the OP. If the question needs to be bumped then they should probably post their own question and link to the old one. Just my $0.05 –  jcolebrand Dec 21 '10 at 13:19
2  
@drachenstern - new users can't comment as they don't have the required rep (50 points) and if they post their own question it's highly likely it will get closed as a duplicate - especially if they link to the original question. I agree more experienced users should comment on the question or even edit it to bump it. –  ChrisF Dec 21 '10 at 13:23
    
"except on very new users" was what I was trying to say for those who can't comment yet, but does it really take 50? I thought it was 15... I think we are in agreement here –  jcolebrand Dec 21 '10 at 13:28
    
@drachenstern - yes it really does take 50 points to be able to comment - check the FAQ ;) –  ChrisF Dec 21 '10 at 13:34
    
@ChrisF: I see, thanks. Maybe new users should be able to upvote questions at least? Or is that still too much of a temptation for sockpuppets? –  j_random_hacker Dec 22 '10 at 7:52
    
@j_random - I think the risk of sockpuppetry is the reason. Though 15's not a huge hurdle. –  ChrisF Dec 22 '10 at 8:50

Clicking yes on the EULA will automatically flag the answer for moderator attention.

This sounds good in theory, but there's a big potential for the mod queue to explode with these flags.

Anyone who is writing a low-quality answer is going to ignore this. Completely. Except possibly if you exploit the UI to make it such that clicking the "expected" button actually cancels the submit... but that's not a very friendly thing to do.

You could implement a subsequent system of rate-limiting, but if you get a lot of unique views, then that will have a limited effect. Not to mention that such a system is complex.

Honestly I don't know what to suggest as an alternative besides setting the hard threshold fairly high.


I also have an heuristic suggestion: answer time relative to question asked time or last modified time. I see a lot of "thank you answers" added much much later than the main activity in a question. Even just that combined with a <100 character criteria is pretty much always a low-quality answer.

share|improve this answer
    
right, that's why this is a specific tab on stackoverflow.com/review called "LATE ANSWERS" –  Jeff Atwood Dec 17 '10 at 18:48
2  
It's not a problem if the number of flags explodes as long as the interface makes it easy to deal with them –  Ivo Flipse Dec 17 '10 at 22:49
    
There are currently 50455 users on Super User, for example, that have a rep threshold between 1-200. Many of these are one-time posters, who never return. This means that a lot of poor quality answers come from these people who have not read the FAQ, do not know the Stack Exchange policies, etc... –  studiohack Dec 18 '10 at 18:26

Collect a database of few hundred good and bad answers and make some machine learning -- good rules will appear automatically. Even better, you can make a challenge out of it -- post a 2/3 of this base with answers and 1/3 without, tell people to predict the missing answers, obtain their results, pick best, give the winner a unicorn-branded mug and deploy the algorithm she/he used. (Small scale version of what Netflix did)

share|improve this answer
    
I like the challenge aspect; it reminds me of Help us choose a sort order for the Stack Overflow homepage, where users could just throw algorithms at the homepage and see how they worked –  Michael Mrozek Dec 17 '10 at 18:12
5  
I think something similar could be done to let posts pop up for moderator attention –  Ivo Flipse Dec 17 '10 at 18:20
    
after some further thought, I think this is only useful for the "Thank You", and "Same Problem" scenarios. The other quality heuristics we use are largely based on characters, entropy, and length not words. –  Jeff Atwood Dec 22 '10 at 7:43
6  
@Jeff This is not a problem; you can add such other descriptions to the challenge data. –  mbq Dec 22 '10 at 8:03

I would be careful using word based heuristics, Example:

Based on your problem, the following solution should work for you:

some code that proves p = np

I had the same problem a few weeks ago, and that's how I solved it! :)

The following link should help you understand your problem better:

http://en.wikipedia.org/wiki/P_versus_NP_problem

share|improve this answer
6  
doubtful that the above would be 1) from a new user 2) short enough to have enough aggregate "bad answer" weight to matter –  Jeff Atwood Dec 17 '10 at 14:47
7  
@Jeff, agreed. I just wanted to illustrate that checking for specific words should have a less significant weight on the quality 'rating' of an answer. –  jjnguy Dec 17 '10 at 14:50

Answers (and questions) containing email addresses are suspicious too.

There is no need to provide your email, and it is easy to filter these out.

Another option is to search for swearwords. I have seen them, but never in good answers. But of course we shouldn't make the clbutic mistake.

And don'nt forget the word noob with possibly a repetition of n, o and b and sometimes ending with s. (like noooooooooobbbbb).

share|improve this answer
5  
In a similar vein, "contact me". Although that's usually found together with the e-mail, I usually see "contact me: piskvor at example dot com". –  Piskvor Dec 17 '10 at 11:11
    
excellent ideas, I meant to include email and forgot –  Jeff Atwood Dec 17 '10 at 17:41
    
That is unless the answer is explaining how to program SMTP, or similar... –  MikeSchinkel Apr 3 '11 at 7:55

Increasing the minimum length will not help much, as bad answers will just be made longer. Using it as a filter to show high rep users answers that may be bad, that could then be voted on to delete, may work very well.

I am now thinking 3 systems need to be in place.

  • A warning that is hard to ignore if a new user posts an answer that the filter thinks is bad.
  • A way for established users to vote to delete bad answers, the number of votes needed should be a lot less if the answer is from a new users and gets a bad number on the filter.
  • A way that high rep users can get a list of “likely bad answers”.

Down voting bad answers doesn’t work, as they don’t get deleted and very often someone else will up vote them as it is not considered nice to down vote an answer from a new user. However using a filter like “HasDownVotes, and IsNewUser” may be a good way to get a list of answers that high rep users then vote on to delete.

share|improve this answer
3  
"warning that is hard to ignore" - alas, we're programmers. We only care about errors ;) –  Piskvor Dec 17 '10 at 11:42
3  
+1 for the ability for 10K+ (or perhaps 20K+) users to be able to vote to delete answers. –  ChrisF Dec 17 '10 at 12:09
2  
people who write bad answers generally can't sustain writing a bad long answer -- in other words, they're so bad that anything of length they write falls apart. That, plus just plain laziness. –  Jeff Atwood Dec 17 '10 at 13:40
    
@Jeff - In general, I think you're right, but this user didn't help your case: stackoverflow.com/questions/4475739/wicket-visibility-problem/… –  Brad Larson Dec 18 '10 at 4:20
    
@brad oh it's so on now –  Jeff Atwood Dec 18 '10 at 4:28
    
@Jeff Atwood: Maybe consider the regex \s30\s*characters? as a low-score flag then? In real questions, powers of 2 should be more common - 32 characters. Edit: Alas, the difference is not really significant - I have expected a difference at least an order of magnitude. Googling "$x character" site:stackoverflow.com gives this: 29: 3 hits, 30: 154 hits, 31: 10 hits, 32: 270 hits, 33: 1 hit (and that's a code golf entry). All hits on the first page look legit (even code golf,grumble;)).Possibly bump the limit to 33, as it seems an under-used number?(Disclaimer: I may be overthinking this) –  Piskvor Dec 20 '10 at 12:11

Also from our earlier days, you can safely ban every answer containing a name or a link to any web hosting company. This is pure spam.

share|improve this answer
3  
Not necessarily. What about "I am running a site with xyz (www.xyzhosting.com) and am encountering this and that error...." –  Pëkka Dec 17 '10 at 12:18
    
Also, "any web hosting" - now that Geocities are gone, that's going to be a looooooooooooooooooooooong list; and I'm willing to bet that some of the names on the list are proper nouns (if I know a hosting named Servers.example, will I get flagged for mentioning servers?) –  Piskvor Dec 17 '10 at 12:49
    
What about comparing an URL in a post of a new user against any URL in the profile of that user? (On Super User also including password recovery tools, PDF printers, PDF-to-Word tools, backup solutions, ...) –  Arjan Dec 17 '10 at 14:57
2  
@Arjan: Since this is a "scoring" algorithm, this could be useful. Although some people link to their blogs in the profile, I guess few would link to the blog as such, instead linking to specific articles. Exact match with URL in profile does have some correlation with spam, as far as I've seen. –  Piskvor Dec 17 '10 at 16:15
    
@Piskvor: A good one. –  user136634 Dec 17 '10 at 18:41

I also find it most irritating when some newcomer puts a footnote to his every answer:

Best regards,
Dork

or

Please get in touch with me,
Dork

or

If you need help let me know,
Dork

This is self-promotion in the most obtrusive way. Should be discouraged.

share|improve this answer
3  
Hmmm, it should be discouraged, yes (and it is, through comments and sometimes through vigilante editing); but while I personally dislike it (and possible solutions have been discussed to death on Meta), I don't think there's a strong enough correlation with answer quality. Let's not try to make this review system into a "detect-every-problem" multimegatool. –  Piskvor Dec 17 '10 at 13:23
10  
yes, this is a totally different topic –  Jeff Atwood Dec 17 '10 at 14:48
1  
I find it more irritating when someone signs an actual answer with "Thanks,[name]". What's there to thank for? The question having been asked so he could answer it? –  BoltClock's a Unicorn Dec 17 '10 at 18:10
7  
@boltclock thanks, BoltClock! –  Jeff Atwood Dec 17 '10 at 18:35
    
@Piskvor: Multimegatool! With a pop-up? –  user150068 Dec 17 '10 at 23:38
    
@Alejandro: YES! AND MARQUEE!! AND BLINK!!! –  Piskvor Dec 18 '10 at 18:57
1  
I might be on my own here, but this doesn't bother me at all, and I don't think it needs policing. -1. –  j_random_hacker Dec 21 '10 at 2:02

I'd add "overuse of the dot" into "character checks"..........using the SuperMegaEllipsis is a pretty strong red flag: \.{4,}

Use of the ellipsis is a ... lesser flag, as it could have valid uses. Double dot..is usually a sign of bad grammar - more serious than an ellipsis, but less than "I buy my dots at a bulk rate" IMHO.

share|improve this answer
1  
Your answer would not qualify - the word "thanks". –  user136634 Dec 17 '10 at 10:30
1  
I'd add "plz" to the checklist –  JoseK Dec 17 '10 at 10:33
    
@Developer Art: Help, help, I'm being oppressed! ;) You are correct, that part should've been a comment. –  Piskvor Dec 17 '10 at 10:37
    
I sometimes use the double dots in summaries, like [..] and so on. –  Arjan Dec 17 '10 at 11:19
1  
@Arjan: Isn't "and so on" supposed to be ... - the ellipsis? Nothing personal, but I am under the impression that using .. is a grammatical error (while 1..31 is valid syntax in some programming languages, but I assume we're talking about English now). –  Piskvor Dec 17 '10 at 11:23
    
:-) Piskvor, the "and so on" was not really part of the example. I use [..] in code snippets, and in quotes from external resources, like "Use of the ellipsis [..] could have valid uses". Is there a better/more preferred way to do that? I even use them when I copy a partial list, like here, where actually I left out items in between too... (Migrated question, I don't show as an editor on Server Fault now.) I never gave it much thought, so any opinion is appreciated! –  Arjan Dec 17 '10 at 11:46
    
@Arjan: Looking at the answer, you're using .. interchangeably with ... - while the meaning of .. is obvious there, you're using it as "means the same thing as ..., and I have one less dot to type". In other words, while it doesn't significantly affect the text, it's IMNSHO a pointless ad-hoc alias of ... (pun not intended). –  Piskvor Dec 17 '10 at 11:54
    
Okay, noted! While we're at the subject, what about the brackets? Is [...] in "In other words [...] it's IMNSHO a pointless ad-hoc alias" alright then? (Hmmm, indeed this might be more about English than about heuristics, and in your proposal you trigger on 4 dots or more, so maybe it's not relevant here?) –  Arjan Dec 17 '10 at 12:11
    
@Arjan: I think that's correct; there are questions on the English site which seems to support this: english.stackexchange.com/questions/2271/… english.stackexchange.com/questions/5972/… –  Piskvor Dec 17 '10 at 12:15
2  
Good tip; playing with the heuristic now and double dots (additional penalty for each set of double dots) is definitely working to identify low quality predicted answers. –  Jeff Atwood Dec 20 '10 at 0:30

contains one or more exclamation points

I've written answers with exclamation marks. They're more than appropriate where they appropriate.

content ends in question mark

I've written such answers. Not because I was making a question out of my answer but to put my answer in a certain literary style.

- contains word "help"
- contains word "test" or "testing"
- contains word "same", "problem"
- contains word "sorry"
- contains word "work", "working"

A huge number of good answers would contain those! You wouldn't believe how many!

Then again, you know better right? Right?

share|improve this answer
6  
re single question mark/exclamation mark: good point. On the other hand, multiple question marks, multiple exclamation marks, and their hybrids, are quite a good heuristic???!? –  Piskvor Dec 17 '10 at 10:28
3  
Q: blah A: You did what!!????? You should have ... :) –  Remou Dec 17 '10 at 12:13
10  
@Remou: That's why these are heuristics for detecting a (possibly) bad answer - not hard rules to automatically delete a bad answer. Something similar happens with the spam flags: not everything that is spam-flagged is actually spam - but most things are. –  Piskvor Dec 17 '10 at 12:36
2  
@Remou: (Also, although Terry Pratchett is an author of fiction, his remark that sanity is inversely proportional to the number of punctuation marks is ... strangely accurate, even in your example) –  Piskvor Dec 17 '10 at 12:45
    
@Piskvor Oddly enough, I have read Terry Prachett, and I thought the smiley indicated a humorous comment. –  Remou Dec 17 '10 at 12:53
    
@Remou: I agree - I meant that as "A: [Just the thought of] what you did [drove me closer to insanity]!!! You should have..." :) Too many levels of meta, I guess. –  Piskvor Dec 17 '10 at 12:58
    
@Piskor Damn you, failed communication in internet!!! You strike again!!! –  Remou Dec 17 '10 at 13:09
    
@Remou: Oh noes!!!! –  Piskvor Dec 17 '10 at 13:17
    
If a stream of exclamation marks contains the substring one or eleven, the penalty should be reversed. :) –  j_random_hacker Dec 21 '10 at 1:50
    
Kyles Mom: What!! What! WHAT!!!!! –  hydroparadise Jun 27 '12 at 14:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .