What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

We've all seen questions where what's being asked is totally ambiguous, typically structured such as:


Help plox I has t3h codes but like it doesn't work. Here it is.

foo(bar){baz};
bing; 
Bong+ bang=naptime;
duck.push(cheese+dooperdooles++); a1= {nerpdedtyderpderp}
duck.push(cheese+dooperdooles++); a2= {nerpdedtyderpderp + 5}
duck.push(cheese+dooperdooles++); a3= {nerpdedtyderpderp}
duck.push(cheese+dooperdooles++); a4= {nerpdedtyderpderp}
duck.push(cheese+dooperdooles++); a5= {nerpdedtyderpderp + 90}

Has lik 159 errors omg help plox need the codes ASAP. Here's teh link: t.co/asdf!@#$-9


So what I propose is, before they can submit questions like this that get closed in a hail-storm of wtfs, we run some checks over it and see if it looks like the question was actually detailed.

Similar in concept to how email clients check for "see attached" when sending an email without attachments to warn you you're about to make an arse of yourself.

Lets devise a method to check for "There's an error" without being specific, and alert them to "Hrmm, are you SURE you were clear about WHAT the error actually was? Does your question look like a pile of garbage too? Is your mother real proud of what you're doing here to these people?"

These questions get closed anyway, but lets beat them to to the chase and fight the problem at the point-of-entry, rather than deal with it later when it's already on the boards.

share|improve this question
10  
I'd like to see the parser that could figure this out. –  Won't Sep 16 '11 at 20:29
    
Me too. But if it's worth-while, it could probably be done. We'd query for closed questions and find something that checks most-of-the-time, check if the user is trusted based on other things... there's likely a strong trend driving all of this we can find behind the data. –  Incognito Sep 16 '11 at 20:31
9  
You forgot to call quack.pop(zingerbotz); That'll get rid of your errors. –  squillman Sep 16 '11 at 20:42
10  
There is nothing ambiguous about Bong + bang=naptime; –  bobobobo Sep 16 '11 at 20:43
    
+1 for the code alone, although I think this is not solvable by a parser –  Pëkka Sep 16 '11 at 21:20
1  
@pekka sure it is if (m/\s+omg\s+/i) then goto FAIL; –  tvanfosson Sep 16 '11 at 21:36
1  
@Gamecat I actually think that the unformatted code was intentional. –  Chris Frederick Sep 16 '11 at 21:50
1  
@Chris It was unformatted in my answer to increase the rage level, I left the fist line as 3 spaces, rather than 4 to make it break. –  Incognito Sep 16 '11 at 22:35
3  
Arguably the most intelligent and effective filter for these sorts of questions is the thousands of pairs of eyeballs belonging to members with enough rep to flag, edit or close. IOW you need an implementation of Bang+ bong = naptime; to ease your frustration rather than trying to dream up an elaborate algorithm. –  slugster Sep 17 '11 at 8:03
    
@pekka the unformatting is incorrect; code that is not formatted as code is already blocked on submission. For this to get through and be a valid example, the code should be formatted. –  Jeff Atwood Sep 17 '11 at 12:06
    
@MarcGravell It might be helpful to get a discussion group going with some people who know the data.stackechange and see if anyone from the maths site has insights into statistical models we might use to find things. –  Incognito Sep 20 '11 at 14:44
1  
There are a preponderance of spelling errors in these posts as well - what about flagging on the ratio of errors/total words (excluding formatted code)? –  Michael Jasper Sep 27 '11 at 16:55

6 Answers 6

up vote 4 down vote accepted

We now have some limited checks in place that prevent excessive code blocks without explanation. I've been reviewing the logs from this, and it was having somewhat mixed results - it was certainly catching a lot of "no context, whatsoever" posts, but it was also catching a number of posts where the code did provide suitable context.

I'm toning down the numbers, to ensure that we don't lose viable questions, while retaining the blocking of posts with almost no context - but this is a tough one, as in a number of cases the code (with a little context) does all the talking.

As an example of a real "question" caught in the trap:

import os

path = '../'
folder = os.listdir(path);

def bla(pattern):
  for book in folder:
      if book[-3:] == 'txt':
          data = open(path+''+book).read()
          if pattern in sanitize(data):
              return True
          else :
              print(book, 'does not contain', pattern)
              return False    
if bla('jane'):
  return 
else:

or

int f(int n)
{
for(int i=2;i<=n;i++)
  {
if(n%i==0)
    {
     return f(n/i);//trying to present n as a product of primes
    }
  }
}

However, there were also some false-positives that I'm seeking to reduce.

share|improve this answer
    
I see you're looking to block posts, would showing an alert with guidelines to things their answer is probably lacking be better, and give them a button they have to "confirm" before "submit"? You don't block anything but we really stress they need a better post. –  Incognito Sep 27 '11 at 11:52
    
the problem there is, most people will always just hit that button –  Marc Gravell Sep 27 '11 at 11:54
    
Maybe there's a second correlation here related to not just the post content but the users-- Do you have a way of seeing what rep levels, days on site, or badges were at time of post? –  Incognito Sep 27 '11 at 13:04
    
@Incognito we are only applying this check to users with < 100 reputation... so all I can say is < 100 ;p No, I haven't explicitly captured that information. I could probably obtain it for all the captured data, but it wouldn't be trivial. –  Marc Gravell Sep 27 '11 at 13:08
    
If the "false positive" requires the user to add a sentence of explanation, so be it -- I don't care if "the code talks", the user should too. Posting just code here is unacceptable under any circumstances for low rep users (or any user, really). –  Jeff Atwood Sep 27 '11 at 15:39
    
@Jeff yes; simply, I've tweaked the numbers based on false positives where there was reasonable context –  Marc Gravell Sep 27 '11 at 15:41
1  
At 4k questions per day I am fine with being more strict on this -- I would much rather err on the side of nagging new users to add more context. –  Jeff Atwood Sep 27 '11 at 15:44

Here's an idea. Stack Overflow is in the unique position of having access to the text of millions of questions as well as a measure of their quality -- the vote totals! This leads to an interesting opportunity.

What you should do is run a month-long contest (similar to the Stack Exchange API contest from a while ago) to write a function that does the following:

Given the raw Markdown text for a question on Stack Overflow, predict the following three attributes:

  • The sign (+ or -) of the question's vote total, after one week

  • The order of magnitude (say, log base 2) of the question's vote total, after one week

  • The eventual close status of the question (closed or not closed), after one week

Let everybody go crazy for a month trying out heuristic approaches, neural networks, bayesian nets -- whatever they want.

Then, run every contestant's algorithm against Stack Overflow's database of questions (older than a week), and compare their output to the actual eventual outcome for that question. If the winner is above some preset accuracy that the community is comfortable with (say, 90%), then give them a nice shiny prize and adopt their technique for throwing up the kind of warning Incognito is talking about.

Addendum: In case it wasn't obvious, the algorithm is not allowed to merely use an SO data dump to look up the value :)

share|improve this answer
1  
We'd also have to see deleted questions (a question is so bad that it gets deleted post-haste). Not sure if this would be allowed since low-rep users can't normally see these. –  Awesome Poodles Sep 17 '11 at 7:16
    
True, that would be needed for algorithms which need a "training set". I'm sure something can be worked out. The reason they're hidden from low-rep users is to shield them from bad content. Showing them is not some sort of reward to high-rep users whose exclusivity has to be maintained. I suspect such questions are accessible from data.stackexchange.com already. –  Adrian Petrescu Sep 17 '11 at 7:30
1  
If you won that "competition" then you may end up with a badge that even Jon Skeet hasn't got, although that would break the laws of physics. –  slugster Sep 17 '11 at 8:00
1  
We have a database of nearly 2m~ Qs, we should have a sufficient dataset somewhere to find it, and tweak accordingly. I'm sure we'll find comments along the lines of "What was the error?" –  Incognito Sep 17 '11 at 14:11

Is the SO stack using Bayesian filtering anywhere? I know FogBugz does some Bayesian filtering to manage incoming emails (e.g. identifying likely sales requests vs. support requests, etc). Something like that could work here.

(That is, we flag messages that fit into this category. The system records common properties that those messages have -- e.g. length, presence of various keywords, presence of various tags (e.g. lots of code), rep score of requester, etc.). The system can them determine a probability that any given new message is likely to be of this type, and the system sets a cutoff probability score to treat the message differently.

share|improve this answer
    
Great idea! How hard could it be? ;) –  Chris Frederick Sep 16 '11 at 22:37
6  
we've experimented with this and it fails miserably -- the difference between "spam" vs "non-spam" is much easier to define than "good programming question" vs "bad programming question" -- it is completely impossible to filter that into bayesian buckets. Feel free to download our creative commons corpus and give it a shot yourself. –  Jeff Atwood Sep 17 '11 at 6:14
    
Hey man, nice shot –  bobobobo Sep 19 '11 at 14:57

The only thing I can think of is this:

If the post has a lot of code with very little explanatory text, it's probably bad.

Which implies we'd look at the ratio of code text posted to non-code text posted. But I'm not entirely sure this would be.. safe. It might be if we set the thresholds very high? Something like 20 thousand lines of code and a single line of text is almost certainly crap.

share|improve this answer
3  
You know, rather than block the post entirely, why not silently flag it or even have Community cast an automatic close vote? –  Chris Frederick Sep 17 '11 at 6:27
    
My gut instinct says this have too many false positives - the average bad programming question will have a similar code to text ratio as the average good programming question - and the outliers are so few that they aren't worth the effort. But I can be wrong –  Pëkka Sep 17 '11 at 8:42
2  
A lot of the 'Baby Ruth bars' I see floating by are just that, a bunch of code predicated or followed by a terse and incomprehensible sentence or two. The problem really is deciding what a 'bunch' should be. –  Tim Post Sep 17 '11 at 10:48
    
@tim this change has been in for a few days now, Marc will document it in an answer soon. –  Jeff Atwood Sep 27 '11 at 8:17

I'm just going to use this post as a journal with what correlations I've tried and what seems to be a dead-end, feel free to join in.

Basically, play around with the data site. We know we have a good trend when 80% of the results match those on closed-questions.

Words in the title

Weak correlation to anything meaningful, the top 85 words account for 50% of questions closed due to not being a real question, for a total of 8634 words. Statistical speaking, if you're outside of the 0.015% most commonly used words, it looks bad for you. That being said, this is probably a bad thing to look for, as "silverlight", "struct", "quicksort" are all in the least-occurring word list.

  • Words in title doesn't have anything that sticks out strongly as a gauge to close questions.

I'll look into body text next up...

share|improve this answer

It seems that the best method would be a weighted total of several different factors:

  • user rep < 100 = 10 points
  • correct spelling/total words < 3/1 = 30 points
  • code/text < 2/1 = 20 points =
  • words in title not in body = 20 points
  • total words < 50 = 20 points

posts > 60/100 points are automatically flagged for review/moderation before they appear on the site or, the ask-er is prompted to fix the question before submitting.

Thoughts?

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .