Can we check for code with little to no explanatory text?

Question

We've all seen questions where what's being asked is totally ambiguous, typically structured such as:

---

Help plox I has t3h codes but like it doesn't work. Here it is.

foo(bar){baz};
bing; 
Bong+ bang=naptime;
duck.push(cheese+dooperdooles++); a1= {nerpdedtyderpderp}
duck.push(cheese+dooperdooles++); a2= {nerpdedtyderpderp + 5}
duck.push(cheese+dooperdooles++); a3= {nerpdedtyderpderp}
duck.push(cheese+dooperdooles++); a4= {nerpdedtyderpderp}
duck.push(cheese+dooperdooles++); a5= {nerpdedtyderpderp + 90}

Has lik 159 errors omg help plox need the codes ASAP. Here's teh link: t.co/asdf!@#$-9

---

So what I propose is, before they can submit questions like this that get closed in a hail-storm of wtfs, we run some checks over it and see if it looks like the question was actually detailed.

Similar in concept to how email clients check for "see attached" when sending an email without attachments to warn you you're about to make an arse of yourself.

Lets devise a method to check for "There's an error" without being specific, and alert them to "Hrmm, are you SURE you were clear about WHAT the error actually was? Does your question look like a pile of garbage too? Is your mother real proud of what you're doing here to these people?"

These questions get closed anyway, but lets beat them to to the chase and fight the problem at the point-of-entry, rather than deal with it later when it's already on the boards.

Answer 1

We now have some limited checks in place that prevent excessive code blocks without explanation. I've been reviewing the logs from this, and it was having somewhat mixed results - it was certainly catching a lot of "no context, whatsoever" posts, but it was also catching a number of posts where the code did provide suitable context.

I'm toning down the numbers, to ensure that we don't lose viable questions, while retaining the blocking of posts with almost no context - but this is a tough one, as in a number of cases the code (with a little context) does all the talking.

As an example of a real "question" caught in the trap:

import os

path = '../'
folder = os.listdir(path);

def bla(pattern):
  for book in folder:
      if book[-3:] == 'txt':
          data = open(path+''+book).read()
          if pattern in sanitize(data):
              return True
          else :
              print(book, 'does not contain', pattern)
              return False    
if bla('jane'):
  return 
else:

or

int f(int n)
{
for(int i=2;i<=n;i++)
  {
if(n%i==0)
    {
     return f(n/i);//trying to present n as a product of primes
    }
  }
}

However, there were also some false-positives that I'm seeking to reduce.

Answer 2

Here's an idea. Stack Overflow is in the unique position of having access to the text of millions of questions as well as a measure of their quality -- the vote totals! This leads to an interesting opportunity.

What you should do is run a month-long contest (similar to the Stack Exchange API contest from a while ago) to write a function that does the following:

Given the raw Markdown text for a question on Stack Overflow, predict the following three attributes:

• The sign (+ or -) of the question's vote total, after one week

• The order of magnitude (say, log base 2) of the question's vote total, after one week

• The eventual close status of the question (closed or not closed), after one week

Let everybody go crazy for a month trying out heuristic approaches, neural networks, bayesian nets -- whatever they want.

Then, run every contestant's algorithm against Stack Overflow's database of questions (older than a week), and compare their output to the actual eventual outcome for that question. If the winner is above some preset accuracy that the community is comfortable with (say, 90%), then give them a nice shiny prize and adopt their technique for throwing up the kind of warning Incognito is talking about.

Addendum: In case it wasn't obvious, the algorithm is not allowed to merely use an SO data dump to look up the value :)

Answer 3

Is the SO stack using Bayesian filtering anywhere? I know FogBugz does some Bayesian filtering to manage incoming emails (e.g. identifying likely sales requests vs. support requests, etc). Something like that could work here.

(That is, we flag messages that fit into this category. The system records common properties that those messages have -- e.g. length, presence of various keywords, presence of various tags (e.g. lots of code), rep score of requester, etc.). The system can them determine a probability that any given new message is likely to be of this type, and the system sets a cutoff probability score to treat the message differently.

Answer 4

The only thing I can think of is this:

If the post has a lot of code with very little explanatory text, it's probably bad.

Which implies we'd look at the ratio of code text posted to non-code text posted. But I'm not entirely sure this would be.. safe. It might be if we set the thresholds very high? Something like 20 thousand lines of code and a single line of text is almost certainly crap.

Answer 5

I'm just going to use this post as a journal with what correlations I've tried and what seems to be a dead-end, feel free to join in.

Basically, play around with the data site. We know we have a good trend when 80% of the results match those on closed-questions.

Words in the title

Weak correlation to anything meaningful, the top 85 words account for 50% of questions closed due to not being a real question, for a total of 8634 words. Statistical speaking, if you're outside of the 0.015% most commonly used words, it looks bad for you. That being said, this is probably a bad thing to look for, as "silverlight", "struct", "quicksort" are all in the least-occurring word list.

• Words in title doesn't have anything that sticks out strongly as a gauge to close questions.

I'll look into body text next up...

Answer 6

It seems that the best method would be a weighted total of several different factors:

• user rep < 100 = 10 points
• correct spelling/total words < 3/1 = 30 points
• code/text < 2/1 = 20 points =
• words in title not in body = 20 points
• total words < 50 = 20 points

posts > 60/100 points are automatically flagged for review/moderation before they appear on the site or, the ask-er is prompted to fix the question before submitting.

Thoughts?