-5

This is the second in a series of posts that we’re using to show (broadly, directionally) some of the potential use cases for generative AI in the Stack Overflow and Stack Exchange environment. The first post was on Yaakov’s title selector demo page.

Today, I bring you another cool demonstration tool: this one is immediately useful for customers of ours, or for anyone using the Stack Overflow for Teams product.

Questions that we frequently get asked when people are evaluating Stack Overflow for Teams are:

  • How big of a problem are repeat questions? In other words, do I really have a problem that Teams can fix?
  • I know I have a problem, but how can I identify which questions should be in Teams to start?

Enter Decipher (final name is still TBD). This tool uses AI to read Slack transcripts and pull out the questions contained within them, then group together the ones that are duplicates or similar.

The use cases here seem to me to be self-apparent - customers can use that list to pre-seed questions and answers, or to make the case internally that a lot of time is wasted on answering the same set of questions in Slack to instead drive aggregation of internal knowledge on Teams. It also calls out when content hadn’t been socialized broadly enough, or distribution wasn’t consistent throughout the company to enable teams to quickly mitigate these knowledge distribution challenges.

I’m curious about other use cases that you can imagine for this tool, or alternative ways we could help the community solve similar problems? Leave your response as an answer below, and I’ll be checking in on them regularly.

In the meantime, we’re thinking about the best way to share the code so you can run this locally - our intention here is to co-create a tool with immediate usefulness, so get those pull requests ready.

3
  • 14
    Please, no bread and circus policy using AI/ML. Instead of making populism based on neural networks, that with us does not work, why don't you improve the search mechanism that has been a repeatedly claimed by the community for a long time? Commented May 4, 2023 at 17:39
  • 2
    Use-case wise, this seems pretty sound– something that AI seems really apt for so far is data aggregation; pulling in irregular inputs (like questions sprinkled across chat logs, for instance) and converting them into something useful is a really neat application IMO. Hearing about the experimentation makes me optimistic, and I really, really hope we get to see benefits of it on the actual platform at some point.
    – zcoop98
    Commented May 4, 2023 at 22:50
  • The gist: "group together the ones that are duplicates" Commented May 5, 2023 at 12:22

4 Answers 4

29

If users are having trouble finding answers on SOfT, why not focus improving the search functionality? It's well known that the best way to search SO/SE is with Google, but that doesn't work on private instances.

Instead of overly convoluted indirect systems to solve XY problems, can we please just focus on improving the already well known problematic systems on the site?

8
  • 11
    As a bonus, the bar for improving search is ridiculously low, on account of search being so bad that it's often unusable. The results in a real search engine will always be better, but as search currently stands, for most real-world search scenarios, real search engines are generally the only option that produce any vaguely good results
    – Zoe
    Commented May 4, 2023 at 15:52
  • 8
    I'm not sure how this relates to the tool described in the question, which is about finding questions that should be on a Teams instance, but aren't. Search can only find existing questions that are on the Team, not questions that are being asked in chat instead of on the Team.
    – Ryan M
    Commented May 4, 2023 at 15:59
  • Related: "I think it'd be more beneficial and productive to work on better search functionality" (my answer to the parent post of this series).
    – starball
    Commented May 4, 2023 at 16:11
  • 6
    @RyanM Part of the implication I got from the video was that this was partially designed for finding duplicates. Also, if so many questions are being asked on slack/MS teams, rather than SOfT, that sounds like there are fundamental problems with the usability of SOfT. SE seems to be focusing on adding new features to solve XY problems, rather than improving the core functionality of their primary product.
    – vandench
    Commented May 4, 2023 at 16:32
  • Re "the best way to search SO/SE is with Google": No, it is broken. For example, on Stack Overflow, try to find the way to output octal numbers in Perl. It is a very basic task, and it was probably asked back in 2008 or 2009, but the search engines refuse to return pages that old. Hints turns up, but not the authoritative canonical question/answer. (Search engine success may vary, of course.) It is an actual example of where ChatGPT works as a super search engine. Commented May 5, 2023 at 12:03
  • 2
    cont' - It isn't just Stack Overflow. Search is broken in general. Commented May 5, 2023 at 12:07
  • 2
    @This_is_NOT_a_forum but the search engines refuse to return pages that old I'm not sure where you got that notion. With google I found this post for printing octal characters from 2013, and this post for printing hex characters from 2009, and that was without even limiting to results from before 2010 (which Google will also happily do).
    – vandench
    Commented May 5, 2023 at 12:11
  • @van dench: Search engine results may vary. The hexadecimal one may happen to work better, but my example was only about octal. It is just one of many, many examples where searching for even the most basic information (that must have been asked on Stack Overflow) only turns up low-scored with low-quality answers (and/or irrelevant) questions from the mid 2010s onward. Eventually the canonical question will be found, but it way too inefficient. Commented May 11, 2023 at 11:08
15

I assume you're wildly experimenting right now and collecting ideas, which is certainly not a bad strategy. But this example is really more of a marketing tool for Teams, and not something that will convince the community about the whole AI initiative.

For the network I think there is one huge obvious problem that would benefit enormously from a good AI solution. That is the detection and handling of duplicates. If an AI tool could just tell users that ask a duplicate question that the answer is already here before they post it, that would be an enormous improvement to the experience for both the asker and the community of answerers and curators.

This is a really old problem, but with the recent advances in LLMs the problem doesn't look as impossible as it used to.

4
8

Do you really need an AI for this? And is an AI really better than what you could achieve in simpler ways?

This tool uses AI to read chat transcripts and pull out the questions contained within them, then group together the ones that are duplicates or similar.

I don't think it would be too hard to parse basic questions out, and really, I don't think it's too hard for the average user to formulate a question or search query on their own either.

Once the question is parsed out / formulated, why not just plug it into the search box?

How much marginal gain is there from building an AI to do this, and how much loss is there from the weaknesses or failures of the AI?

And in a similar vein to what @van dench said in their answer post, why not just focus on teaching users to use the search functionality of the site (and potentially also of popular search engines)? Knowing how to use search operators in the site search bar and popular search engines can provide a lot of long term value (speaking from experience).

It could also help to give sites control over their search stemming. For related discussion, see is:q [search] stemming on MSE and is:q [search] stemming on MSO.

1
  • just focus on teaching users to use the search functionality of the site (and potentially also of popular search engines)--yes
    – philipxy
    Commented May 6, 2023 at 8:11
4

Well, I'd wonder if this would work as a better Clippy bot.

Traditionally, we've discouraged people from asking questions on chat; there's even a rather amusing Easter egg on Super User linked chatrooms. An optional SE chatbot that pulls posts from across the network when someone asks a question seems... pretty useful potentially.

5
  • I know the chat is all public and open, but I prefer to think nobody is monitoring and harvesting every message from it… I know it’s possible, but it’s quite a creepy thing, so I prefer to imagine it doesn’t happen. A chatbot based on «AI»/ML that finds questions for us, without being prompted? Sounds like a data harvester. Not that SE would need a chatbot for it. They already harvest all the data for use by the chat service itself. I just don’t want it used for other purposes. Commented May 4, 2023 at 15:59
  • 4
    'Chat' in the question seems to be referring to Slack, not SO/SOfT chat.
    – Alex Poole
    Commented May 4, 2023 at 16:10
  • 1
    "Traditionally we've discouraged people from asking questions on chat" - I'm not sure about the rest of the network, but there are chat rooms in SO chat that exist for Q&A. anime.se also refers users to chat for certain off-topic questions
    – starball
    Commented May 4, 2023 at 16:20
  • @Andreasdetestscensorship there's literally an Easter egg that converts how do I questions to search queries 😅 Commented May 4, 2023 at 23:00
  • @AlexPoole well yes. That's entirely intentional - since it's a somewhat neglected internal resource Commented May 4, 2023 at 23:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .