27

I think Stack Overflow could be a very interesting AI test bed for answer bots.

Obviously it would be too much work for a single person, but maybe a university or corporate research group...

I'm thinking specifically about the IBM DeepQA bot.

It seems like it might be able to at least ID and answer the most easy "what is the definition of:"-style questions on Stack Overflow, which is why I was wondering whether it had been done before.

Do you think such a bot would be capable of maintaining a positive reputation on Stack Overflow? Or would wrong answers pummel it into the group?

migrated from stackoverflow.com Dec 6 '10 at 18:50

This question came from our site for professional and enthusiast programmers.

  • 40
    I'm not sure you realize it but Jon Skeet is a bot ;-) – mjv Dec 6 '10 at 18:32
  • do you have IBM's resources? – Mauricio Scheffer Dec 6 '10 at 18:33
  • 6
    It only needs to handle regex and pointer questions, and reply "is this homework" to anything worded as a assignment and it will deal with most questions :) (Ha, ha, only serious) – The Archetypal Paul Dec 6 '10 at 18:33
  • @Mauricio Let's pretend that I do. Do you think it is possible? – John Shedletsky Dec 6 '10 at 18:34
  • BTW, with its broad and "meta" nature, this question has good chances of being closed. On this occasion I'd like to plug the upcoming stackexchange site Artificial Intelligence (area51.stackexchange.com/proposals/6607/artificial-intelligence) where this type of question [well... maybe expressed in more specific ways] would have more chances of surviving. If you haven't yet committed to this site, I encourage you to do so. – mjv Dec 6 '10 at 18:37
  • 2
    The beta version of such a bot could be just the standard search for duplicates... – MPelletier Dec 6 '10 at 20:39
  • I've considered trying to bot some of the more mundane moderator activities, but that would be really hard to explain in the event of a catastrophe. An answer bot seems much safer. – Bill the Lizard Dec 6 '10 at 20:53
  • 1
    @Bill but what about a janitor bot that has the same rights as, say, a 2k user? That can e.g. vote to close but not close unilaterally? To top it off, written in some standardized language that people can suggest edits and patches in. Oh, and flag bad decisions. Would be really interesting to see that in action - could be a project for a AI-related University course or a lab.... – Pëkka Dec 6 '10 at 21:33
  • @Rejoice: That would definitely be interesting to play with. I can't decide if it would be more useful on newer sites with fewer 2K users, or older sites with more established standards. – Bill the Lizard Dec 6 '10 at 21:38
  • @Bill SO would be the most interesting field for a bot I think, because the topic is so very technical. I can see a bot failing much more often for, say, gardening related questions than questions on jQuery – Pëkka Dec 6 '10 at 21:41
  • @Rejoice: That's true, an advanced enough bot could test out its own programming answers. Not true for gardening or cooking. :) – Bill the Lizard Dec 6 '10 at 21:44
  • @Bill awww. The real contest will then be "who can make the bot crash by feeding it an undigestible code sample?" :) – Pëkka Dec 6 '10 at 21:45
  • In order to be real AI and not just a bot, it should handle natural language. There is a whole computer field about this issue, where last works on declartive paradigm makes it a must to have tool. – user150068 Dec 6 '10 at 22:29
  • Did you mean to say "pummel it into the ground"? – Glen_b Aug 8 '16 at 2:52
  • This question has been up for 7 years and no one's linked xkcd 810, yet? Time to fix that. – Michael Feb 8 '18 at 17:49
22

I doubt whether a bot could give consistently decent answers even in this day and age except for the very simplest of questions.

What might work, however, is AI serving as "janitor-bots".

  • Vote to close as duplicate - e.g. judging from a network of related questions of which many were closed as dupes

  • Vote to close as spam

  • Ask for clarification when code is missing (but understand when it is not missing)

  • Suggest missing tags

  • Suggest re-formatting when user forgets code blocks

I would find it fascinating to see a very small number of (officially licensed and displayed) bots doing janitorial work - always within the limitations a normal user has, too - and seeing how well they work out.

  • Well, this answer bot had consistency, and was quite good. It gave answer way better that "decent". – Ashish Ahuja Aug 6 '16 at 16:31
15

Yes. There was an answer bot created by the University of Antwerp which ran on Stack Overflow. It mainly focused on Git questions.

This is the paper which gives us more information of the bot. The technical report is found here.

The bot is currently closed.

It had answered lots of questions, with many answers actually valid.

  • 6
    seems like the bot should have been voting to close as dupe since its methodology was to use the answers to duplicates of the asked question. The rep it earned seems very low. Why was it banned? – Kate Gregory Aug 6 '16 at 16:35
  • 1
    @KateGregory: Because it was a bot answering questions it should have been flagging/voting to close? – Nathan Tuggy Aug 6 '16 at 18:33
  • 1
    I'm curious to know if a mod made a decision, or of all the downvotes it got trigged a "we are no longer accepting" ban. – Kate Gregory Aug 6 '16 at 18:54
  • @KateGregory Regarding your rep question: Well the second account which disclosed that it was a bot got very less rep because of the ban by mods. The first account which acted like a human got 300 rep, which I think is great for a bot. Regarding your ban question: I'm not sure. We have to wait for a mod – Ashish Ahuja Aug 7 '16 at 6:19
  • what account was the bot run under? – user340193 Nov 29 '16 at 17:08
  • @MarkYisri Tay, Mike, Ricardo, Jake, Joey (they are in a specific order). Go through the technical report I just linked in my answer to see why one account was better than the other. – Ashish Ahuja Nov 30 '16 at 2:52
1

It probably would be possible, given enough effort.

However, SO has effective anti-bot defenses.
You would need a special interface from the SO team.

  • 1
    Yeah, to be clear, I was more interested in the theory of it than the practice. – John Shedletsky Dec 6 '10 at 18:33
1

I don't think (given today's technology) it would be able to maintain a positive reputation unless it's extremely selective in what it tries to answer - in which case it may only answer 1-2 queries/week. We already have something similar...it's called Google. If people are posting questions where the first Google response ISN'T the answer, then I doubt someone's going to be able to do better (unless they had more resources than Google). If people are posting questions where the first Google response IS the correct answer, then they are nimwits & their questions should just be voted down.

  • 4
    I'd disagree somewhat - I think a bot could get a positive reputation, at least until a recalc, by playing cheap. It just has to use the "fastest gun in the west" strategy, which an AI is uniquely suited for, on common repeat questions. With this strategy it doesn't even need to use Google - it just looks for questions by new users, with no responses, for which its regex finds a similar (closed) question from its copy of the data dump. It then pastes the accepted answer from that question. – David Dec 6 '10 at 19:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .