Present and future of bots on Stack Exchange

Question

There are discussions on Meta already for some time now about the potential usage of bots on Stack Exchange (SE). The earliest I could find goes back to 2009, but there have been others [link], [link].

Here’s an interesting one about having a bot that tries to answer “easy” questions. The scenario is that (almost) duplicate, simple questions are asked time after time, despite SE’s efforts to prevent this by listing related questions at posting time. This is where the bot could be of service: it could quickly retrieve and post an answer from a previous duplicate (or with smarter Artificial Intelligence - AI) even from elsewhere on the web, like a user manual or a mailing list discussion). I imagine this might save some review time for moderators, and perhaps users would also enjoy not having to answer over and over again to the same question. Besides, answers would be very fast, which should make everyone happy if they are also of good quality.

This idea of “janitor” bots can have more use cases besides answering “easy” questions (see also discussion here), like suggesting missing tags and re-formatting what should have been code blocks, to name a few.

Yet, despite a long time since these discussions, better Machine Learning Classifiers (ML) and AI over the years, and tons of academic research around SE’s websites, it doesn’t seem that any such bots are among us (except maybe Jon Skeet). As part of a research project at the university, I am trying to understand why this is the case.

For this reason I’d like your opinion on the following:

1. Do you consider a bot “harmful” for the SE communities?
   • If yes, would you consider the bot harmful even if
     • it operates in an automatic manner only in limited scenarios (e.g. specific tags)
     • it operates in a semi-automatic manner (namely with the supervision of a moderator who decides for posting or not the answer).
     • it operates only for a limited amount of time in a small community (e.g. here).
2. Would you consider it too risky to give a bot a try?
3. Would you accept an answer provided by a bot?
   • If not, why ? (e.g. you don’t trust replies from machines)
4. Would you consider useless a “sloppy” answer by a bot, even if it is provided in just a few seconds? Answers may be not perfect, but they may link to other relevant threads that can help the asker to fix the problem without waiting extra time for a human response.

Answer 1

The usefulness of a bot has already been proven. And it's not the only one bringing problems to peoples' attention.

The real question you seem to be asking is whether a bot making posts would be harmful. If the bot's doing something practically Watson-like, then sure, that's great. If it can stitch together a valid, correct answer, and do it all in well-Markdowned, coherent English, I'd accept its answers as fast as it could provide them. One of the main principles around here is generally understood to be that you never, ever vote/flag/act based on who posted something, but instead on the content of the post.

The problem is, what you're proposing is content I'd be downvoting/flagging/voting to delete from anyone. Grabbing answers from prior duplicates is absolutely not allowed, to the point where advice to that effect is part of the community FAQ. You said that "perhaps users would also enjoy not having to answer over and over again to the same question", but we don't want users to do that in the first place. If the same question is posted over and over, we want them closed as duplicate questions, which is why Stack Exchange has functionality specifically for doing that.

On the other hand, if you (or anyone) can make a bot that identifies duplicate questions, has a lower false-positive rate than humans, and votes or flags to close them (or possibly brings them to the attention of a human monitoring the bot, so they can take action manually), that's a great idea. As I mentioned at the start of this post, there's precedent for similar automation, and when that post came up, the response was generally positive.

Well, what about copying/linking to information "from elsewhere on the web, like a user manual or a mailing list discussion"? Still not cool. Just blindly copying it is blatant plagiarism, and link-only answers are about as accepted as duplicate answers.

The ideas of "suggesting missing tags and re-formatting what should have been code blocks", though? Those would be great, as long as the bot was right often enough that it's beating humans.

---

TL;DR: Users will be judged by their actions, whether they're human or not. Make sure a bot's actions are ones the community is okay with, and we'll probably be okay with our new robotic overlords bots participating.

Answer 2

In the case of a duplicate-finding bot, this would be extremely useful if it works well. But it shouldn't take the form of an answer, the proper response by a real user would be to comment, flag or vote to close as duplicate.

What I'd do with such a bot would be to feed it into a dedicated chat room and get a few active users to check the suggestions by the bot and vote to close as duplicate, if the bot is correct. This would give you some data to evaluate the bot while helping to actually close questions as duplicates with some human supervision.

If such a bot turns out to work well, it could lead to some improvements in the duplicate question process.

Answer 3

If you are a researcher on the AI side of computer science, and you have got a duplicate-finding bot, please do the following:

• put the source on GitHub
• make it work with StackExchange data dumps
• post a question/answer describing your training and testing samples and hypothesis testing procedure and testing results.

Stack Exchange is in dire need of such software.

If you have no such code and no plans to write it I'd reckon that your question is rather speculative and not constructive.

Answer 4

Do you consider a bot “harmful” for the SE communities?

A bot can only be useful to flag, not to answer, since you can't just copy or link another post (that is now how we work). When the bot only flags, you should have a very high threshold or it is likely to be flag banned within a few tries.

Would you consider it too risky to give a bot a try?

That is up to you. It is risky, and I think the SE team is better capable of doing such things (you might want to ask why they didn't do it yet...) and in fact should be the people organizing this. They are better capable of altering the UI (if necessary) to incorporate this and to measure it is working according to plan.

Would you accept an answer provided by a bot?

Would you consider useless a “sloppy” answer by a bot, even if it is provided in just a few seconds? Answers may be not perfect, but they may link to other relevant threads that can help the asker to fix the problem without waiting extra time for a human response.

No, bots shouldn't answer. The problem is that their answers will always be low-quality, link-only answers or blatant copies of a previous post. Both not the type of answers we want.

Answer 5

I believe I've shown that a bot performing janitorial tasks is useful. There are also several other such bots around the network. These janitorial bots are utilized to great effect in keeping low quality "stuff" from appearing for long across the network.

I am very much in favor of automation, where it is appropriate. I don't think providing answers is such a location, at this time. To have an automated system answer programming questions and not simply copy/paste previous answers AND provide an explanation of why it's answers work, would be a considerable research investment. Remember that good answers aren't simply a code block that runs. It is an explanation of why this code block works versus why the original didn't, or an explanation of why this code block is more efficient than another. It is the explanation that makes a good answer.

As others have mentioned, an area that would make an amazing bot, would be duplication detection. Such a bot would fit very well into the janitorial role the ones mentioned above fall into. On Stack Exchange, I think this is the role that bots will fall into most often. Unlike reddit, or other sites, we don't need (or want) bots that provide additional context (wikipeida bots, image transcribers, link fixers, etc). Our goal is focused on providing high quality answers to user questions. A large portion of "high quality" is moderation/janitorial work.