An example of a generative AI tool: Suggesting better question titles

Question

With all the conversation around generative AI and our early explorations into it, I thought it might be useful to post some screenshots that show examples of the type of early experiments that we’re considering as a technological set of solutions to address long-standing challenges we’ve had on the site. We probably should have led with this, or included it as a part of my post about Prashanth’s blog post, and for that I take responsibility. We’re learning as we go, and I’ll try not to make that mistake again.

One example of a potential use for generative AI is in creating better titles for questions. These screenshots show examples of how AI might add value for the community here. I’d like to say very clearly that these are early, proof-of-concept implementations of a testing page (not actually integrating it into any asking workflow yet), and we will be seeking community feedback on workflows like this. They are intended to demonstrate the line of thinking that we are taking on this. It’s important to say that we will try to include in this and other generative AI tools that may be rolled out in the future mechanisms for community members to give feedback on output and to help with model training, in order to ensure knowledge creation and content quality is maintained before and after any changes.

In the first screenshot, we see an actual user-generated question with a sub-par title that has code completely inline as text. The second box contains new suggested titles, the third box has the question’s tags, and the fourth box has the body of the question.

The tool retrieves the question from a url using the Stack Overflow api, and returns a few suggested better titles.

---

The second screenshot works the same way, but in this case the code is in a code block. Note that the titles returned are still valid and incorporate the fact that the code is CSS into them.

---

And a third screenshot - in this one, the code is mixed - some inline and some within a code block. Note that the potential titles that are given actually incorporate part of the error message that is given in the body.

---

Again, this is just some preliminary work, and we are doing explorations and research on how to utilize AI/ML in ways that are promoting a better user experience for things like question asking, search, duplicate detection, and more.

This is the type of initial experiment that we’d like to carry out on Stack Overflow - I’m hard put to see how the result here is anything but a net positive for the knowledge repository. We end up with better titles, which hopefully drive content discovery and better quality answers. We know that title creation is difficult for new askers, and this is a low-friction way to make that process easier for both askers and moderators alike. We hope to proceed with the same types of goals for other similar experiments that we may run, and are aiming to keep community members involved as we do so.

Next post in this series: AI/ML Tool examples part 2 - Extracting questions from Slack transcripts and finding similar/duplicate SO for Teams questions.

Answer 1

I'm going to go out on a limb here and suggest that this is a good example of the use of generative AI being integrated into Stack Exchange proper. I'm still going to take the position that you guys should be careful in this exploration and (obviously) be prepared for pushback if some other feature explorations end up going a bit too far for experienced curators' comfort, but I do think this is a solid idea.

The title is easily the most neglected field when new users draft a new question. I still kinda suck at drafting question titles, but I can recognize a garbage one when I see one. That's one of the reasons why, over on the Staging Ground Beta test, I suggested we add a "Please revise your title" comment template, and another user suggested that it links to How do I write a good title?, and it's turned out to be quite the useful template. Sometimes I can salvage a crappy title myself, but I do at times find myself unable to fully grok what a user is looking for, and want to shift the responsibility onto them to write a proper title. If generative AI could help in this area, I personally don't think this is a bad idea.

We probably should have led with this, or included it as a part of my post about Prashanth’s blog post, and for that I take responsibility. We’re learning as we go, and I’ll try not to make that mistake again.

Look... You've gotta understand where a majority of Meta-frequenting-users are coming from: They're relentlessly fighting against an abundance of crappy, AI-generated answers on their favorite sites. Moderators in particular have to sift through more garbage than ever as a result of ChatGPT and other AI tools gaining popularity. They're building advanced tools to try and make detecting and eliminating AI-generated content easier on themselves because they're utterly overwhelmed. There's absolutely going to be some trepidation about some form of generative AI being released on the site they're trying to purge it from existence on. Please take the pushback you got from the CEO's blog post with that in mind.

we are doing explorations and research on how to utilize AI/ML in ways that are promoting a better user experience for things like question asking, search, duplicate detection ...

Oh boy do we need that. Finding duplicates is quite unrewarding, difficult, and leads to squabbles between gold badgers on Stack Overflow all the time. Ensuring that users see potential answers before they even post a question would be great, because as the site grows older, the amount of information that may already answer a new question becomes even more numerous. Lifting the burden of finding a duplicate off curators' shoulders would be a great step forward.

We hope to proceed with the same types of goals for other similar experiments that we may run, and are aiming to keep community members involved as we do so.

That last bit makes me very hopeful. I mean, the fact that you pulled the curtain back a bit here on what y'all are working on so we could get an idea of where you feel generative AI could be useful is great, because we really needed to see that you guys weren't allocating resources to stuff we're going to outright hate. This is a departure from "release and announce" that I'm hoping to see more of, especially on such a hot-button issue as AI is right now.

Answer 2

I'm... cautiously optimistic. I see the value here, however there's also a potential for this to cause problems when people do a poor job of evaluating whether or not the title they chose matches what they're asking for.

Take example 2; The suggested titles are certainly better than "doesn't work", if we assume the titles actually describe their problem... but I'm not so sure that they do because the user hasn't really presented enough in the question to support anything more than "doesn't work". If you don't know why it doesn't work, you can't assume it's because they added classes to elements, or whatever it is they actually did since that doesn't seem to be part of the screenshot.

A very large percentage of the questions we get on a daily basis are like example 2.

Converting useless titles into useless titles with proper English isn't ideal... but at least it'd be a tool to help improve the questions that aren't this hopeless? I'd certainly appreciate having something like this for editing a question title once the question is improved.

---

Put another way, I fear that this may put us in a situation where question titles can be inaccurate because we're asking users who do not understand the problem they are asking about to choose a "recommended" title from a list of AI recommendations... potentially adding incorrect context/information to their question. It's one thing for a title to be... lacking information, it's another entirely for wrong information to be added to it.

Answer 3

I'm optimistic about this. Writing good titles is a skill / something one gains an intuition for after having a better understanding of what Stack Exchange is about (knowledge base), so newer users tend to do poorly at it. If the AI ends up being good at its job enough of the time (off the top of my head, maybe 90%?), I'd be all for it.

Promote the relevant Help & FAQ pages too, please!

I do hope that this isn't treated as the ultimate, single solution though. I'd rather the system teach the user to fish than always just give them the fish: I'd want this to go hand in hand with a UI that gives more serious encouragement to read the Help Center pages- /help/how-to-ask in particular. We also have a post here on MSE that I often link people to: How do I write a good title?.

The other reason why this is important is because askers don't just need guidance with writing better titles. They often need guidance with the rest of their question post as well. I literally just came across a post with a two-word title, and the rest of the post was desperately in need of more detail, in which case I don't think an AI for suggesting titles (presumably based on the rest of the question post (what else?)) could have done much with. Garbage in, garbage out.

Suggestion: Experiment with models on a per-site basis

I wonder if a different model should be built for each network site. No strong reason in particular. I'm just thinking it might be that one size does not fit all. Perhaps worth experimenting with.

Suggestion: Give training feedback based on title edits

How would you build the model and give it initial data on what's considered a good title? Would you use score as a proxy? That might have some base in general, but not all poorly-scored questions have bad titles and not all well-scored questions have great titles. Perhaps you could somehow take into account titles that get edited by users and incorporate info about the reputation of the user who did the edit. Ex. if a high-reputation user edits the title, you can use the initial title as an example of a less-desirable title, and the new one as a more desirable one. That might backfire in cases where the edits to the title are very minor, so the "drastic-ness" of the edit could be good to take into account as well: more-drastic being taken to mean more likely that the previous title was less good.

I know that being a high-rep user does not necessarily make one better at writing titles- there can be a danger of switching to technically-correct terminology that newbies wouldn't think to use, which can actually sometimes be detrimental to the searchability of the post (if they switch all occurrences in the title and body), but having lots of subject-matter-expertise and site experience on Stack Exchange tends to increase one's ability to write descriptive and non-ambiguous titles (ones that make it clear how a question is different from other similar questions), which both are quite valuable (actually- what I personally think are the two most important things in a title).

---

I'm glad better search and duplicate detection are on the todo-list! Those are exactly the two things I'd expressed hope to see more work done on in the first post in this series.

I've also given more of my thoughts in my answer to the MSO post, leaning more into programming-related things and challenges with generality/ambiguity in questions.

Answer 4

Let me just comment on

We end up with better titles...

and not on the layout of the proposed feature. I wonder how you have defined "better" in the training of the AI tool? Hopefully better doesn't only translate to a higher likelihood of visits (click bait optimization) but to something like appropriateness of the content and usefulness in searches.

In my mind, the ideal title would strike a balance of summarizing the content as specifically as possible and length on the other hand. Hopefully, this definition overlaps with whatever you mean by "better titles". Additionally, I worry that if the content of the question is insufficient, it's in general impossible to generate even a good title. The title can only be generated from the content, there might be problems even earlier in the asking work-flow.

Finally, have you thought about using that for automatic duplicate detection? A title is a summary of the content. If distinct questions would have a very similar or equal generated titles, they might likely be duplicates.

In any case, you need to decide what happens if the generated titles already exist on the site.

Answer 5

I honestly went from suspecting the recreational use of illicit substances to 'OOH SHINY!' - and I feel like this a great example of both poor messaging to the community and great messaging.

I remember previously - there was a team dedicated to data science on SE, some use of data to 'enhance' the user experience , and a previous machine learning project called Providence.

Is the planned work on generative AI going to be building on that?

Answer 6

Regardless of utility I remain concerned about transparency, traceability, and accountability in deploying generative "AI". The big looming question for me is:

What is the training dataset?

I am well aware that the resale/licensing of pre-trained generative ML models (as well as the practice of calling them "AI") is starting to obscure the concept of training datasets in the ML space, but it remains true that there is a dataset behind every model. If SE were to license one such pre-trained model and be unable to account for every single input datum, I would be extremely uncomfortable with using the site. It would feel like laundering the reputation of the legally/morally questionable practice of reckless data hoovering used to train many of these ML models, through the tacit approval of the reputably conscientious SE userbase.

Now, if this were a model solely trained on SE content (or a larger dataset with at least as many assurances about its composition of open-licensed or otherwise legitimate inputs), from start to finish, that would be a different matter. But generative ML models are expensive to train, and with this post not mentioning where the model is coming from, I have concerns about where SE is planning to source this model.

Answer 7

Why is it not even generating titles that are actually questions? Only 3 out of the 9 examples are questions, the rest are statements. (And one of the 3 doesn't have a question mark, so it could be parsed as a statement too - "How to X" is a common title for tutorial blog posts.) These titles look like bug reports, tutorials, rants, but not questions!

Unless you can make it generate only actual questions, this should not be put into common use.

(I know some sites are soft on whether question titles need to actually be questions, but surely any automated system like this needs to generate exemplary titles, not borderline ones.)

Answer 8

Hammer all the nails

It seems to me like a license was acquired for using this, and now you have this awesome hammer in search of nails.

Title suggestion isn't so bad. Perhaps it could analyze existing posts though for "does this answer your question", although that isn't really something the GPT does very well, the transposition of facts where the question is different but still asking the same thing.

The thing is, titles need to be accurate, and they will directly influence page rank and such (of which Stack Overflow is falling behind of late -- that is a much different conversation though).

Let's focus on question closure

Why not use this in a place where some arm waving and extra explanations are actually useful? In question closure. Questions are closed at a constant pace, and having a prompt that explains what happened to the asker would be one such place that this technology would excel. It doesn't need to be entirely 100% correct so long as it cites what it was trying to explain; namely that your question was closed for asking about GoDaddy settings and that isn't on topic here. It could provide links for canonical or help center content as well as just expanding on what is on topic or why third party settings aren't on topic, etc.

Answer 9

Can we please not have an automated tool that produces bad English? One of the example "good" titles is:

How to use redis-om .matches function instead of equals to search for entries containing a string?

This is missing the before redis-om. You can't "use redis-om matches function", you need to "use the redis-om .matches function". More importantly, it is using "how to" as a question!

"How to foo" is a statement, it is not a question. You cannot ask "how to foo?", that isn't English. This is the problem with generative AI models: they don't know anything, and since this is a very common anti-pattern on the internet, the AI seems to think that "how to" is a good way of starting a question. It isn't. "How can I foo?" is a question, "how to foo" is not.

Now, if you at least don't add a question mark at the end, "how to foo" can be a valid title as demonstrated by the hundreds of "how-tos" out there. However, "how to foo?" is as nonsensical as "this is answer?".

So, if we are going to do something like this, can we at least try to get it right? If even the toy examples generated for a post like this are wrong, this really doesn't fill me with confidence that it will actually be an improvement. The idea is sound, and this is indeed an area where a good AI could really help, but let's just be sure that it actually works and not just throw AI around hoping it will stick.

Answer 10

I dislike this for one simple reason:

it doesn't really fix the underlying problem of an OP not knowing how to write a good title.

Or put another way...

I cannot trust any LLM to implicitly generate "good" or "coherent" titles, nor can users who aren't able to normally write passable titles actually validate the titles that are generated.

Now, I don't necessarily disagree with the notion that giving someone some title suggestions can help inspire them to come up with a better question title, but...if you just give them a title, they're going to just take that and run.

If you have faith that the average question asker on Stack Overflow wouldn't do something like this and would learn from this, you haven't met the average question asker yet.

(I mean, c'mon. We see this crap come up all the time in that some poor OP was deluded into thinking that the AI response they saw - and didn't realize it was from an AI - was a good response that they could use. Don't give us more headaches.)

Basically - this gives someone a solution, and they run with it. If they're met with animosity over this, or edits over this, then yet more friction ensues because, well, the system gave them this title, and the system clearly knows better than the curators of the site, so why should the curators bother editing it out?? /s

Note: There are others who share my concern even if they're cautiously optimistic about the overall effect this'll have. Can someone who leans solely on AI to help determine if their title is good or bad? How do we evaluate that in context? What if the title is edited anyway?

But more seriously, this is the crux of the problem with AI on Stack Exchange. People just wanna help. The challenge is knowing what kind of help you can and cannot realistically provide.

Helping people write better questions or frame their problem in a more concise manner is explicitly not something that AI can help with. All AI does here is give them a free pass, which - ironically - defeats the purpose of all of this.