6

There has been a fair bit of discussion about how "bad" code in answers can be dealt with - particularly as, once it is published it can be hard to remove. However "best answers" seem to have a degree of subjectivity surrounding them, so moderating for the avoidance of "bad" answers is not as obvious as it seems. What is not subjective though, is the long term effect of answers in the repository which contain potentially harmful code.

There are many systems for testing scripts and I wonder whether there might be any usefulness in adding an automated filter to all published code - to check that it is runnable and "safe" - rather like https://runnable.io/ though not, perhaps as comprehensive.

It could be assessed for potential attack vectors:

http://www.technicalinfo.net/papers/StoppingAutomatedAttackTools.html

and compared to a database of known bad practices (for example the misuse of a function like $.eval). Such a system could flag "bad" code in answers before it ever saw the light of day and reduce the frustration of OPs who get offered code snippets which will not run and cause even more confusion.

I don't think anyone deliberately posts "bad" code but perhaps such a tool might increase consciousness and help where perhaps there is no method available to test code at the time of posting in the hope that it will be OK.

The race to answer new questions and, accordingly, gain reputation has been discussed, and benefits of "quick and dirty" answers do exist, it seems: Fastest Gun in the West Problem but possibly a tool like this might help all of us improve even the code we produce in a hurry to give that quick answer.

Clearly, with two-and-a-half million users, manual moderation by the community itself seems a problematic expectation going forward and automation could, perhaps, be a part of a solution to that. Should we peer review questions before publishing them? and Add Peer-Reviewed Answer

Would it be feasible or desirable in the interest of safe coding?

  • This is equivalent to the Stopping Problem. Undecidable, unfortunately. – Deer Hunter Jan 10 '16 at 20:22
  • @DeerHunter - do you have a link to any discussions of the Stopping Problem? - It all started from Rachel's post here for me meta.stackoverflow.com/questions/258085/… – Steve Jan 10 '16 at 20:29
  • If you have useful heuristics, you could submit them to authors of the SmokeDetector bot. There's an article about the Halting Problem on Wikipedia. – Deer Hunter Jan 10 '16 at 20:35
  • Wow SmokeDetector looks pretty powerful! I am not suggesting a system which will test if code is fully runnable as much of it will be snippets, but there might be a half-way-house that could put up an "are you sure you want to post this?" when it contains parseable errors and possibly something like SmokeDetector with a database/heuristics for the known "bad" stuff. The Halting Problem would at least be removed from that equation, though clearly there is only so far it can go. This already happens at lower levels for detecting post formatting for code in a way. – Steve Jan 10 '16 at 20:51
  • @DeerHunter SmokeDetector is actually only for detecting spam, not mere low-quality posts. – The Guy with The Hat Jan 10 '16 at 22:57
  • @TheGuywithTheHat - I noticed that it classified some code as possible obscenity, but the fact that it could do that seemed to suggest it may be possible to "educate" it, as I think DeerHunter was implying if I understood correctly, by his reference to heuristics. If we can detect viruses using those, perhaps that kind of technology might be brought to bear? – Steve Jan 11 '16 at 0:02
7

The problem with automating code evaluation is that code is, ultimately, written for humans to read. If you try to make a computer read something that was written for humans, it can do it, but it can't always make sense of it.

It doesn't take a massive stretch of the imagination to imagine that there could quite easily be some code that requires context to either make sense, or be error-free. Or what about multiple code blocks? If you try to analyse every code block separately, you'll come up against issues with that: take this for an example:

If you're trying to foo the bar, your best bet is to define your own function to do it, like this:

function foo(bar) {
    return bar.foo();
}

If you've only got a pseudo-bar, then you can add this line to the start of the method body:

bar = bar.getFullBar();

If you analyse that second code block on its own, it'll show up as an error because bar isn't defined anywhere. You'd have to analyse all the code blocks together, and also together with the question code (because answers often make use of variable names in the question, with no explicit definition).

You've also got the possibility of false positives and false negatives, and potentially, you get legal issues. If you start telling people that all code posted on the site is analysed for possible attack vectors or exploits before it's made visible, and people rely on that, then if you forget to disclaim responsibility then you can get sued for it.

Don't get me wrong, I think automated code evaluation would be an awesome addition to the sites - but it has to be done right, and that's difficult.

  • Quite so! I am curious to know how it might work. Thanks. – Steve Jan 10 '16 at 23:00
  • I think you could make a good case, in fact, that this kind of analysis could be the single killer feature behind an entire startup, its raison d'etre, the way Stack has "vote-sorted Q&A wikis" as its core mission. – Nathan Tuggy Jan 10 '16 at 23:35
  • 1
    @NathanTuggy agree completely. Which is also the reason that Stack shouldn't try to do it without heavy outsourcing. – ArtOfCode Jan 11 '16 at 0:09
  • @ǝpoɔɟoʇɹɐ I agree with you personally there and have noticed that a lot of discussion is focused on what "we" can do about problems where outsourcing might provide the scale and coordination of professional resources which I believe it would need to get a solution of this nature. There must be plenty of members who would have such resources in their professional worlds. – Steve Jan 11 '16 at 12:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .