89

With all the conversation around generative AI and our early explorations into it, I thought it might be useful to post some screenshots that show examples of the type of early experiments that we’re considering as a technological set of solutions to address long-standing challenges we’ve had on the site. We probably should have led with this, or included it as a part of my post about Prashanth’s blog post, and for that I take responsibility. We’re learning as we go, and I’ll try not to make that mistake again.

One example of a potential use for generative AI is in creating better titles for questions. These screenshots show examples of how AI might add value for the community here. I’d like to say very clearly that these are early, proof-of-concept implementations of a testing page (not actually integrating it into any asking workflow yet), and we will be seeking community feedback on workflows like this. They are intended to demonstrate the line of thinking that we are taking on this. It’s important to say that we will try to include in this and other generative AI tools that may be rolled out in the future mechanisms for community members to give feedback on output and to help with model training, in order to ensure knowledge creation and content quality is maintained before and after any changes.

In the first screenshot, we see an actual user-generated question with a sub-par title that has code completely inline as text. The second box contains new suggested titles, the third box has the question’s tags, and the fourth box has the body of the question.

Screenshot of a potential title suggestion tool, with code inline

The tool retrieves the question from a url using the Stack Overflow api, and returns a few suggested better titles.


The second screenshot works the same way, but in this case the code is in a code block. Note that the titles returned are still valid and incorporate the fact that the code is CSS into them.

Screenshot of a potential title suggestion tool, with code in a code block


And a third screenshot - in this one, the code is mixed - some inline and some within a code block. Note that the potential titles that are given actually incorporate part of the error message that is given in the body.

Screenshot of a potential title suggestion tool, with code mixed and inline


Again, this is just some preliminary work, and we are doing explorations and research on how to utilize AI/ML in ways that are promoting a better user experience for things like question asking, search, duplicate detection, and more.

This is the type of initial experiment that we’d like to carry out on Stack Overflow - I’m hard put to see how the result here is anything but a net positive for the knowledge repository. We end up with better titles, which hopefully drive content discovery and better quality answers. We know that title creation is difficult for new askers, and this is a low-friction way to make that process easier for both askers and moderators alike. We hope to proceed with the same types of goals for other similar experiments that we may run, and are aiming to keep community members involved as we do so.

Next post in this series: AI/ML Tool examples part 2 - Extracting questions from Slack transcripts and finding similar/duplicate SO for Teams questions.

24
  • 33
    While I oppose the current usage of AI worldwide and the "jumping on the wagon", I do like this use, well done! Plus, thanks for the honesty and learning from mistakes, totally non trivial. So even though I was going to downvote on spot, I upvoted. (And that's rare. ;)) Commented Apr 19, 2023 at 16:33
  • 37
    I will literally pay Stack Overflow money to start putting the title last in the /ask forms. One simply cannot know a good title without first knowing the body's contents, be it a question, a scientific journal, a news article, or a novel.
    – TylerH
    Commented Apr 19, 2023 at 16:34
  • 11
    @TylerH how is that even possible? The one asking the question should know what the question is about, otherwise the question won't be very good anyway. Commented Apr 19, 2023 at 16:41
  • 3
    @TylerH : see this. Though I'm not really impressed with the answer, the first comment under the question is at least one good reason not to.
    – CDR
    Commented Apr 19, 2023 at 16:51
  • 18
    @ShadowTheSpringWizard The title should be a summary of the question in the body. Getting a question fully fleshed out and complete makes summarizing it easier. A lot of people don't understand that it's a summary, and use the title field, which is first, to introduce their question, often describing the context in which they encountered it, but not talking about the problem itself.
    – Ryan M
    Commented Apr 19, 2023 at 17:28
  • 39
    "We probably should have led with this, or included it as a part of my post about Prashanth’s blog post" The blog post should just have been concise & clear instead of rambling ChatCEO marketspeak.
    – philipxy
    Commented Apr 19, 2023 at 17:46
  • 3
    "anything but a net positive" This is tautology--if you develop useful things they will be useful. You do not know that any particular thing will be useful. The issue is the opportunity cost of feature development. Poor titles generally come with poor posts. The details of feature behaviour matters very much. I am doubtful that AI will ever do better than a policy change & a red box that pops up after title entry that says STOP. MAKE SURE THIS IS A PRECISE CONCISE CLEAR VERSION OF YOUR QUESTION & ALSO NOT APPLICABLE TO A DIFFERENT QUESTION. OTHERWISE YOUR POST WILL LIKELY GET CLOSED.
    – philipxy
    Commented Apr 19, 2023 at 18:00
  • 4
    @philipxy people don't read, so doubt this will help much. Commented Apr 19, 2023 at 18:02
  • 7
    Please use text for what can be given via text. Like the text content of those images.
    – philipxy
    Commented Apr 19, 2023 at 18:24
  • 1
    @Chindraba That doesn't solve the problem of putting the title first, in addition to what starball said about poor UX. Not to mention very few people overall use Twitter (and even fewer are good at it).
    – TylerH
    Commented Apr 19, 2023 at 20:21
  • 7
    At a minimum, there should be some checks that it follows title recommendations (such as not prefixing the question with the tag, which it does suggest in the first and second example) and that the title meets the quality check heuristics (these seem to, but I'm not sure every title will)
    – Erik A
    Commented Apr 20, 2023 at 8:58
  • 6
    The second one has "when certain elements are removed" but the issue is occurring when certain classes aren't removed, not when certain elements are removed. The first one looks OK, not sure about the third one because part of the question is cut off. It may be that "when certain elements are removed" makes the title look more like a good Stack Overflow question title in general, but "does this look like a good title?" is too much easier to measure than "is this a good title for this question?". There is a risk of optimising for what's easy to measure rather than what's actually right.
    – kaya3
    Commented Apr 20, 2023 at 23:28
  • 6
    I like this application. Down to earth, realistic, practical. This is how progress is made. Commented Apr 22, 2023 at 8:19
  • 9
    The problem, of course, is that the "AI" doesn't understand anything about the questions it's generating titles for. There is no adjusting it to ensure it only produces "better" titles if you don't understand why it's generating the titles it is. At best this'll be released, knowing that some large percentage of the time it will produce incorrect titles, expecting users to decide which ones are correct. The very users who are struggling to understand their own problem.
    – Kevin B
    Commented Apr 25, 2023 at 18:34
  • 4
    There's no need to use AI. Things to improve user experience as a whole has already been suggested years ago. Unfortunately, it was ignored, as usual...
    – hkotsubo
    Commented May 17, 2023 at 0:31

10 Answers 10

64

I'm going to go out on a limb here and suggest that this is a good example of the use of generative AI being integrated into Stack Exchange proper. I'm still going to take the position that you guys should be careful in this exploration and (obviously) be prepared for pushback if some other feature explorations end up going a bit too far for experienced curators' comfort, but I do think this is a solid idea.

The title is easily the most neglected field when new users draft a new question. I still kinda suck at drafting question titles, but I can recognize a garbage one when I see one. That's one of the reasons why, over on the Staging Ground Beta test, I suggested we add a "Please revise your title" comment template, and another user suggested that it links to How do I write a good title?, and it's turned out to be quite the useful template. Sometimes I can salvage a crappy title myself, but I do at times find myself unable to fully grok what a user is looking for, and want to shift the responsibility onto them to write a proper title. If generative AI could help in this area, I personally don't think this is a bad idea.

We probably should have led with this, or included it as a part of my post about Prashanth’s blog post, and for that I take responsibility. We’re learning as we go, and I’ll try not to make that mistake again.

Look... You've gotta understand where a majority of Meta-frequenting-users are coming from: They're relentlessly fighting against an abundance of crappy, AI-generated answers on their favorite sites. Moderators in particular have to sift through more garbage than ever as a result of ChatGPT and other AI tools gaining popularity. They're building advanced tools to try and make detecting and eliminating AI-generated content easier on themselves because they're utterly overwhelmed. There's absolutely going to be some trepidation about some form of generative AI being released on the site they're trying to purge it from existence on. Please take the pushback you got from the CEO's blog post with that in mind.

we are doing explorations and research on how to utilize AI/ML in ways that are promoting a better user experience for things like question asking, search, duplicate detection ...

Oh boy do we need that. Finding duplicates is quite unrewarding, difficult, and leads to squabbles between gold badgers on Stack Overflow all the time. Ensuring that users see potential answers before they even post a question would be great, because as the site grows older, the amount of information that may already answer a new question becomes even more numerous. Lifting the burden of finding a duplicate off curators' shoulders would be a great step forward.

We hope to proceed with the same types of goals for other similar experiments that we may run, and are aiming to keep community members involved as we do so.

That last bit makes me very hopeful. I mean, the fact that you pulled the curtain back a bit here on what y'all are working on so we could get an idea of where you feel generative AI could be useful is great, because we really needed to see that you guys weren't allocating resources to stuff we're going to outright hate. This is a departure from "release and announce" that I'm hoping to see more of, especially on such a hot-button issue as AI is right now.

2
  • 2
    "and another user suggested that it links to How do I write a good title?" *does a curtsey :)
    – starball
    Commented Apr 21, 2023 at 2:42
  • 1
    Gotta give credit where it's due!
    – Spevacus
    Commented Apr 21, 2023 at 2:43
29

I'm... cautiously optimistic. I see the value here, however there's also a potential for this to cause problems when people do a poor job of evaluating whether or not the title they chose matches what they're asking for.

Take example 2; The suggested titles are certainly better than "doesn't work", if we assume the titles actually describe their problem... but I'm not so sure that they do because the user hasn't really presented enough in the question to support anything more than "doesn't work". If you don't know why it doesn't work, you can't assume it's because they added classes to elements, or whatever it is they actually did since that doesn't seem to be part of the screenshot.

A very large percentage of the questions we get on a daily basis are like example 2.

Converting useless titles into useless titles with proper English isn't ideal... but at least it'd be a tool to help improve the questions that aren't this hopeless? I'd certainly appreciate having something like this for editing a question title once the question is improved.


Put another way, I fear that this may put us in a situation where question titles can be inaccurate because we're asking users who do not understand the problem they are asking about to choose a "recommended" title from a list of AI recommendations... potentially adding incorrect context/information to their question. It's one thing for a title to be... lacking information, it's another entirely for wrong information to be added to it.

3
  • 5
    Perhaps the exposition by @Philippe could include an example of "sorry, your post doesn't seem to contain enough information to extract a title from".
    – tripleee
    Commented Apr 20, 2023 at 3:47
  • 2
    @tripleee -There are obviously a ton of different exceptions that we would need to include in the final design before rolling anything out. I'm not sure which have been built yet and which haven't (as I said above, this is an early design and one that I pushed to make public in order to have an example to show, because I thought it was important. This is nowhere near code-complete or ready to integrate.)
    – Philippe StaffMod
    Commented Apr 23, 2023 at 22:35
  • @Philippe Sure; thanks for the reply. I'm thinking it might clarify the scope of the planned feature if you could include this corner case.
    – tripleee
    Commented Apr 24, 2023 at 3:40
21

I'm optimistic about this. Writing good titles is a skill / something one gains an intuition for after having a better understanding of what Stack Exchange is about (knowledge base), so newer users tend to do poorly at it. If the AI ends up being good at its job enough of the time (off the top of my head, maybe 90%?), I'd be all for it.

Promote the relevant Help & FAQ pages too, please!

I do hope that this isn't treated as the ultimate, single solution though. I'd rather the system teach the user to fish than always just give them the fish: I'd want this to go hand in hand with a UI that gives more serious encouragement to read the Help Center pages- /help/how-to-ask in particular. We also have a post here on MSE that I often link people to: How do I write a good title?.

The other reason why this is important is because askers don't just need guidance with writing better titles. They often need guidance with the rest of their question post as well. I literally just came across a post with a two-word title, and the rest of the post was desperately in need of more detail, in which case I don't think an AI for suggesting titles (presumably based on the rest of the question post (what else?)) could have done much with. Garbage in, garbage out.

Suggestion: Experiment with models on a per-site basis

I wonder if a different model should be built for each network site. No strong reason in particular. I'm just thinking it might be that one size does not fit all. Perhaps worth experimenting with.

Suggestion: Give training feedback based on title edits

How would you build the model and give it initial data on what's considered a good title? Would you use score as a proxy? That might have some base in general, but not all poorly-scored questions have bad titles and not all well-scored questions have great titles. Perhaps you could somehow take into account titles that get edited by users and incorporate info about the reputation of the user who did the edit. Ex. if a high-reputation user edits the title, you can use the initial title as an example of a less-desirable title, and the new one as a more desirable one. That might backfire in cases where the edits to the title are very minor, so the "drastic-ness" of the edit could be good to take into account as well: more-drastic being taken to mean more likely that the previous title was less good.

I know that being a high-rep user does not necessarily make one better at writing titles- there can be a danger of switching to technically-correct terminology that newbies wouldn't think to use, which can actually sometimes be detrimental to the searchability of the post (if they switch all occurrences in the title and body), but having lots of subject-matter-expertise and site experience on Stack Exchange tends to increase one's ability to write descriptive and non-ambiguous titles (ones that make it clear how a question is different from other similar questions), which both are quite valuable (actually- what I personally think are the two most important things in a title).


I'm glad better search and duplicate detection are on the todo-list! Those are exactly the two things I'd expressed hope to see more work done on in the first post in this series.

I've also given more of my thoughts in my answer to the MSO post, leaning more into programming-related things and challenges with generality/ambiguity in questions.

5
  • 3
    "How would you build the model and give it initial data on what's considered a good title?" Is it a model or is it ChatGPT that's been given a prompt like, "Take the following question body and write a specific title that's less than 150 characters long…"?
    – Laurel
    Commented Apr 19, 2023 at 18:04
  • 1
    Fully agree, myself I am writing comment "Please don't paste screenshots, use formatted text for code and error messages. See Why should I not upload images of code/data/errors?" almost every day! Commented Apr 21, 2023 at 6:10
  • 1
    @WernfriedDomscheit, I totally agree with the sentiment that there are definitely interface elements that need to be tied with such an implementation. As I called out in the initial text, this was for a test page, purposely disconnected from the existing interface to show only this technology. Relevant help and FAQ pages would certainly be in the interface when in a production workflow.
    – Philippe StaffMod
    Commented Apr 23, 2023 at 22:32
  • 3
    @Philippe "Relevant help and FAQ pages would certainly be in the interface when in a production workflow." How prominently, and with what kind of description text? The current ask wizard hides the link to /help/how-to-ask in a single word, and the regular Ask Question UI puts it in a collapsable menu in the sidebar. There's no inline mention at all of what the benefits of reading and applying the guidance found there are: better clarity and form of posts -> better voting reception and ease of answering
    – starball
    Commented Apr 23, 2023 at 22:36
  • 2
    @starball - no idea. As I said, they haven't been designed yet. But I'll be sure that the teams in question take this as a follow-up note for when they do the design. "early exploration outside of workflow", remember.
    – Philippe StaffMod
    Commented Apr 24, 2023 at 2:24
12

Let me just comment on

We end up with better titles...

and not on the layout of the proposed feature. I wonder how you have defined "better" in the training of the AI tool? Hopefully better doesn't only translate to a higher likelihood of visits (click bait optimization) but to something like appropriateness of the content and usefulness in searches.

In my mind, the ideal title would strike a balance of summarizing the content as specifically as possible and length on the other hand. Hopefully, this definition overlaps with whatever you mean by "better titles". Additionally, I worry that if the content of the question is insufficient, it's in general impossible to generate even a good title. The title can only be generated from the content, there might be problems even earlier in the asking work-flow.

Finally, have you thought about using that for automatic duplicate detection? A title is a summary of the content. If distinct questions would have a very similar or equal generated titles, they might likely be duplicates.

In any case, you need to decide what happens if the generated titles already exist on the site.

4
  • 1
    Interesting point about taken titles. But if a title is already taken, I think that probably means the machine didn't do its job well (not descriptive/disambiguating enough), or the asker didn't do their job well (actually a duplicate)
    – starball
    Commented Apr 21, 2023 at 9:30
  • Clickbait is effective. "Software developer breaks silence on string search method that shocks the entire industry" Commented Apr 28, 2023 at 11:23
  • A related problem is that the title space is already highly congested. Existing questions with titles like "read from two files" often turn out to be about how to do an inner join of two tab-separated files or something, for example.
    – tripleee
    Commented Apr 28, 2023 at 12:58
  • @tripleee One would need to make the titles more specific then. They could potentially become very long. Example "Get column values of a DataFrame if values of same column name matches and the name of the column matches row value of another DataFrame [duplicate]" Unfortunately it still was a duplicate (of a question with a very short title called "Pandas Merging 101"), which by itself isn't a question but a tutorial. Commented Apr 28, 2023 at 15:25
7

I honestly went from suspecting the recreational use of illicit substances to 'OOH SHINY!' - and I feel like this a great example of both poor messaging to the community and great messaging.

I remember previously - there was a team dedicated to data science on SE, some use of data to 'enhance' the user experience , and a previous machine learning project called Providence.

Is the planned work on generative AI going to be building on that?

4
  • 1
    Providence is still a system that's in use in our production environment, and a huge part of our ecosystem. The work that we're talking about here doesn't build directly on Providence, however, to the best of my knowledge. (Caveat: I'm not an architecture/infrastructure expert, by any definition, and I could be misunderstanding. But I don't think so.)
    – Philippe StaffMod
    Commented Apr 24, 2023 at 4:49
  • Ah, oddly enough, I remember that question, and honestly that was very much an intentionally clickbaity title. I'm curious how you generated the 'new' title though. Commented Apr 27, 2023 at 22:58
  • @JourneymanGeek - The newest release of Chat-GPT through my OpenAI account. I fed it the content of the page and asked it to produce a title for the post. If you want in chat or somewhere else I can give you the actual set of prompts.
    – Travis J
    Commented Apr 28, 2023 at 18:16
7

Regardless of utility I remain concerned about transparency, traceability, and accountability in deploying generative "AI". The big looming question for me is:

What is the training dataset?

I am well aware that the resale/licensing of pre-trained generative ML models (as well as the practice of calling them "AI") is starting to obscure the concept of training datasets in the ML space, but it remains true that there is a dataset behind every model. If SE were to license one such pre-trained model and be unable to account for every single input datum, I would be extremely uncomfortable with using the site. It would feel like laundering the reputation of the legally/morally questionable practice of reckless data hoovering used to train many of these ML models, through the tacit approval of the reputably conscientious SE userbase.

Now, if this were a model solely trained on SE content (or a larger dataset with at least as many assurances about its composition of open-licensed or otherwise legitimate inputs), from start to finish, that would be a different matter. But generative ML models are expensive to train, and with this post not mentioning where the model is coming from, I have concerns about where SE is planning to source this model.

4

Why is it not even generating titles that are actually questions? Only 3 out of the 9 examples are questions, the rest are statements. (And one of the 3 doesn't have a question mark, so it could be parsed as a statement too - "How to X" is a common title for tutorial blog posts.) These titles look like bug reports, tutorials, rants, but not questions!

Unless you can make it generate only actual questions, this should not be put into common use.

(I know some sites are soft on whether question titles need to actually be questions, but surely any automated system like this needs to generate exemplary titles, not borderline ones.)

8
  • 3
  • 5
    Most of the time things like "How do I" are noise & the question is implicitly obviously the title prefixed with something like, "What is the solution to my getting" Even your example of an alleged question title "How to X" is an example of that because it is a sentence fragment & not a question but English headline/title conventions allow us to see the intended question.
    – philipxy
    Commented Apr 20, 2023 at 3:33
  • 2
    On the front page as of right now, 38% of the posts have a question mark somewhere in them. It's not like it's impossible that more than 60% are junk at any given moment, but seriously, I would say the majority of good questions that I have seen do not have an explicit question mark in the title.
    – tripleee
    Commented Apr 20, 2023 at 3:49
  • @tripleee Do you mean here on Meta.SE? Meta sites do run differently, where bug reports, tutorials and blogs are allowed. If you mean on Stack Overflow or another site, well some sites are softer on whether titles need to be phrased as questions or not. I'm just trying to point out that some sites are not soft like that, and this system really should be producing questions if it's going to be at all applicable across the network. Commented Apr 20, 2023 at 5:24
  • 3
    I meant Stack Overflow's main page stackoverflow.com. Many questions are phrased like "Foobar causes blitzle when it shouldn't" rather than "Why does Foobar cause blitzle?"
    – tripleee
    Commented Apr 20, 2023 at 6:26
  • 4
    Titles needn’t be questions. They need to describe what the question is about. On Code Review, for example, the title must never be a question, if it is a question the post most likely doesn’t belong on the site. Commented Apr 21, 2023 at 3:28
  • 1
    @Cris Sure, each site has its own expectations for titles. Buy CR is an outlier in this regard. Most sites do actually concern questions, and most prefer the titles to be actual questions. Commented Apr 21, 2023 at 3:53
  • 2
    I'm not sure I'd say "most sites prefer X" without more support– SE is a big, varied place. That said, I do agree that, as presented, optimizing for certain kinds of title formats over others could potentially cause friction if it's not considered properly; it's a good call-out of a possible blind-spot.
    – zcoop98
    Commented Apr 21, 2023 at 15:31
4

Hammer all the nails

It seems to me like a license was acquired for using this, and now you have this awesome hammer in search of nails.

Title suggestion isn't so bad. Perhaps it could analyze existing posts though for "does this answer your question", although that isn't really something the GPT does very well, the transposition of facts where the question is different but still asking the same thing.

The thing is, titles need to be accurate, and they will directly influence page rank and such (of which Stack Overflow is falling behind of late -- that is a much different conversation though).

Let's focus on question closure

Why not use this in a place where some arm waving and extra explanations are actually useful? In question closure. Questions are closed at a constant pace, and having a prompt that explains what happened to the asker would be one such place that this technology would excel. It doesn't need to be entirely 100% correct so long as it cites what it was trying to explain; namely that your question was closed for asking about GoDaddy settings and that isn't on topic here. It could provide links for canonical or help center content as well as just expanding on what is on topic or why third party settings aren't on topic, etc.

7
  • 1
    I'm not really comfortable with the idea of system-level explanations for why something was closed being generated by an AI. What if what the AI generates is wrong?
    – starball
    Commented Apr 27, 2023 at 21:51
  • 1
    The problem with what we have right now (standardized closure messages with Help Center links) is mostly (I assume, with no data to back up the assumption) a problem with the readers, who don't click the links to read the Help Center pages, which I trust far more than what some AI could string together. (The one evidence I have that people don't read the Help Center pages linked in the close notices is all the reddit complaints, where the people complaining just don't seem to get why their question is closed).
    – starball
    Commented Apr 27, 2023 at 21:52
  • @starball - It sounds like you are not very familiar with the closure system and messaging, especially at Stack Overflow. The problem is not with readers, although you could argue they did not read in advance. The issue is that once the closure occurs they immediately begin seeking answers to why, which often includes posting on mSO. A majority of times those posts are closed as duplicate of the overall explanations, or, of the post explaining why they are now question banned.
    – Travis J
    Commented Apr 27, 2023 at 22:07
  • 1
    All of this explanation and interaction would be well served by an AI prompt. The prompt would not be generated by AI, it would be regurgitated. Regurgitation is what these GPT implementations do best, really it is all they do well. The AI could be solely trained from the canonical FAQ posts, tag wikis, and help center articles. This would make it a much more narrow focus on what it could provide, it could be closely gated in order to not generate anything beyond knowledge, and it would be a way to offload the work of handling these mSO questions about closure all the time.
    – Travis J
    Commented Apr 27, 2023 at 22:09
  • 1
    "It sounds like you are not very familiar with the closure system and messaging, especially at Stack Overflow" why / how so? (I don't think so). "although you could argue they did not read in advance" the close notice only appears after closure. The least they could do is read it then. "The issue is that once the closure occurs they immediately begin seeking answers to why" all they need to do is read the closure notice and the linked Help Center pages. "Regurgitation is what these GPT implementations do best" seems like they're pretty good at assertively hallucinating too.
    – starball
    Commented Apr 27, 2023 at 22:17
  • In the why/so category... a lot of the guesses you make have been discussed at length on mSO and MSE, and you don't seem to allude to any of the discourse, decisions, or changes made to address any of the concepts you reference. The rest of your response is in that same vein. Please, if this is something you are interested in, do some research on the topic.
    – Travis J
    Commented Apr 27, 2023 at 22:20
2

Can we please not have an automated tool that produces bad English? One of the example "good" titles is:

How to use redis-om .matches function instead of equals to search for entries containing a string?

This is missing the before redis-om. You can't "use redis-om matches function", you need to "use the redis-om .matches function". More importantly, it is using "how to" as a question!

"How to foo" is a statement, it is not a question. You cannot ask "how to foo?", that isn't English. This is the problem with generative AI models: they don't know anything, and since this is a very common anti-pattern on the internet, the AI seems to think that "how to" is a good way of starting a question. It isn't. "How can I foo?" is a question, "how to foo" is not.

Now, if you at least don't add a question mark at the end, "how to foo" can be a valid title as demonstrated by the hundreds of "how-tos" out there. However, "how to foo?" is as nonsensical as "this is answer?".

So, if we are going to do something like this, can we at least try to get it right? If even the toy examples generated for a post like this are wrong, this really doesn't fill me with confidence that it will actually be an improvement. The idea is sound, and this is indeed an area where a good AI could really help, but let's just be sure that it actually works and not just throw AI around hoping it will stick.

9
  • It looks like .equals, not equals. Commented May 22, 2023 at 15:11
  • 2
    Yes, it should probably be biased (skewed) for a certain type of output (if that is possible). Something like "Provide the output as a regular English question or as a statement. But never, never, never, never ever the "How to ...?" form" Commented May 22, 2023 at 15:20
  • 4
    I think you're missing that what's important in titles isn't "proper" English, it's coherent English. In my view, the titles should be optimized to convey information in a concise and coherent manner, and buffering "the"s are often superfluous to this end in titles. There's just not that much space, and the title you've quoted, frankly, conveys the meaning to me just fine without being 100% "proper".
    – zcoop98
    Commented May 22, 2023 at 16:00
  • 1
    @zcoop98 sorry, but the one thing that generative AI models consistently get right is grammar and style and we are going to use one that can't even do that much? If it is so bad that it can't even catch basic grammar and usage problems, how can we trust that it will get anything else right? And I have spent hours over the years editing and correcting English in posts across the network, do we really need an automated system that will add to that? I understand you don't care about clarity but others do, so let's not impose artificial errors on top of human ones.
    – terdon
    Commented May 22, 2023 at 16:03
  • 1
    @terdon Do not twist my words. I said, verbatim, "titles should be optimized to convey information in a concise and coherent manner", which conveys exactly the opposite of "don't care about quality". You're also conflating "grammatically imperfect English" with "the model is so bad", which just seems ridiculous to me. My point is that if that titles are semantically coherent and meaningful, properly describing the post they're attached to, then they serve their purpose.
    – zcoop98
    Commented May 22, 2023 at 18:10
  • 1
    You are absolutely welcome to nit the grammar and ask for grammatically perfect titles from the model; I just don't agree that grammar perfection is more important to "quality" than semantic meaning, nor that the widely accepted practice of dropping "the"s in technical speech to shorten phrases is so egregious a flaw that it makes the model unfit for use. I feel that what matters most is that people can interpret the titles and that they have the correct meaning, and this seems to be the case upon first impression.
    – zcoop98
    Commented May 22, 2023 at 18:13
  • 3
    @zcoop98 woah, sorry! I had no intention of twisting anything! On the contrary, I was trying to say you have every right to not be a nitpicking language pedant, but just remember that some of us are, and this hits us exactly where it hurts. And I said you don't "care about clarity" not quality. My point is that "how to foo?" is not semantically coherent, but I will grant you that it is meaningful in that I can understand it. But I can also understand "add how can I add integers two" but that doesn't mean it is a good title. But yes, that one is clearly far worse than the example in the OP.
    – terdon
    Commented May 22, 2023 at 18:21
  • Also, the missing the is a very small part of my answer, most of which focuses on my personal pet peeve of "how to foo?" as a question. I must have made hundreds of edits over the years removing question marks from "how to" titles. However, in either case, what's the point of having an automated system if that automated system isn't producing correct English, the one thing that generative AI models can be relied on to do well. If we're going to use one that can't even do that right, then I have even less faith that it will do anything at all right.
    – terdon
    Commented May 22, 2023 at 18:22
  • 1
    @terdon Apologies if my response was overly harsh! I think I read frustration into your response that wasn't really there, that's definitely my bad. Your clarification is super reasonable and on point; stepping back a little myself, I don't think your lack of faith is unreasonable the way you've presented it, and I can definitely sympathize with the editing away bad titles. I think I overestimated how clear I was being too; I was synonymizing "clarity" in titles with "quality", and I didn't make that plain enough, or give room for titles being "clear" but still bad, which is very possible.
    – zcoop98
    Commented May 22, 2023 at 19:32
1

I dislike this for one simple reason:

it doesn't really fix the underlying problem of an OP not knowing how to write a good title.

Or put another way...

I cannot trust any LLM to implicitly generate "good" or "coherent" titles, nor can users who aren't able to normally write passable titles actually validate the titles that are generated.

Now, I don't necessarily disagree with the notion that giving someone some title suggestions can help inspire them to come up with a better question title, but...if you just give them a title, they're going to just take that and run.

If you have faith that the average question asker on Stack Overflow wouldn't do something like this and would learn from this, you haven't met the average question asker yet.

(I mean, c'mon. We see this crap come up all the time in that some poor OP was deluded into thinking that the AI response they saw - and didn't realize it was from an AI - was a good response that they could use. Don't give us more headaches.)

Basically - this gives someone a solution, and they run with it. If they're met with animosity over this, or edits over this, then yet more friction ensues because, well, the system gave them this title, and the system clearly knows better than the curators of the site, so why should the curators bother editing it out?? /s

Note: There are others who share my concern even if they're cautiously optimistic about the overall effect this'll have. Can someone who leans solely on AI to help determine if their title is good or bad? How do we evaluate that in context? What if the title is edited anyway?

But more seriously, this is the crux of the problem with AI on Stack Exchange. People just wanna help. The challenge is knowing what kind of help you can and cannot realistically provide.

Helping people write better questions or frame their problem in a more concise manner is explicitly not something that AI can help with. All AI does here is give them a free pass, which - ironically - defeats the purpose of all of this.

23
  • 21
    I don't follow. Is the idea that it's not teaching them to summarize their problem concisely, but instead concisely summarizing it for them? If so: who cares? That's true of community members fixing the title, too, but this is way less work for the community, and would help reduce the influx of questions with atrocious titles like "Python simple calculator not working" or "I create a calculator using html,css, javascript,so after run the javascript part not working,What is the problem with this code?" that overwhelms the community's ability to fix them. It's hard to get worse than those.
    – Ryan M
    Commented Apr 19, 2023 at 16:57
  • 1
    @RyanM: It only numbs the pain of someone not knowing how to concisely express their question. It doesn't actually fix it. Like I said I don't necessarily mind this specific use case but I don't want to delude anyone in thinking that we've "improved" something here. All we've done is plastered over the problem in enough of a way that we (as in the rest of the site) don't feel the pain anymore.
    – Makoto
    Commented Apr 19, 2023 at 16:58
  • 18
    If the idea is that it's not capable of the task of improving the writing, it sure seems like it can, based on those examples. Even if it doesn't always produce a perfect result, the bar here is low. A lot of question titles are just really bad (the examples in my previous comment are real Stack Overflow question titles), and meaningfully improving that would be a pretty big improvement in terms of post quality. The title is arguably the most important part of a post to get right; reducing the pain there is a real improvement, even if it's not a complete solution.
    – Ryan M
    Commented Apr 19, 2023 at 17:01
  • 4
    I disagree with your position because it's up to each individual to decide whether or not to benefit from the stimuli received from the community, whether it's through direct comments from users or an improved algorithm for question titles. It's up to the user to recognize individually whether frequent corrections to their question titles are a personal flaw or limitation. Collectively, the goal of SE communities is not only to help the question author, but to help the readers who seek that knowledge over time, and a tool that improves titles benefits them. Commented Apr 19, 2023 at 17:18
  • 15
    My main question after reading this answer is: how can you levy this complaint against this, but not levy it against the community manually fixing titles? Is it not exactly the same in terms of what's learned by OP?
    – zcoop98
    Commented Apr 19, 2023 at 17:20
  • 4
    @AugustoVasques: At some point, if an individual decides not to take advantage of the stimuli provided by the community, the individual is restricted or rate-limited in a fairly automated and aggressive fashion. Ask enough bad questions and you can't ask them anymore, for instance. Doesn't matter if the title is generated by AI or not in that case.
    – Makoto
    Commented Apr 19, 2023 at 17:27
  • 2
    @zcoop98: Somewhat fair criticism; I'll justify it by saying that it's about the timing. An OP has all the time in the world and no backlash to ask a question and formulate it in the best way possible, and also to summarize their question in a title as clearly as possible. Once they hit the submit button, that question is now live, and if the question was actually aight but had a bad title, those of us who care will want to fix that up in an attempt to help the OP.
    – Makoto
    Commented Apr 19, 2023 at 17:29
  • 8
    @zcoop98: I'm still of the impression that the ML doing this would take a critical step out of the question writing process of "hey, are you sure what you're asking makes sense?" Instead of having to think through the question they're asking in a way that makes things better for all parties involved, the ML gives them something and they implicitly trust that something without really understanding the other half of the issue. Writing good titles is still the burden of the question asker. If all we do is give them what to use, we have not improved anything.
    – Makoto
    Commented Apr 19, 2023 at 17:30
  • 1
    Makoto, looking solely at the individual, I agree with your comment, to the point where even a user blocked by the system may contribute to the community in a way that restores their privileges within it. My point is that our focus should not be solely on the individual asking the question, but on the community created over time by those who benefit from that content. Commented Apr 19, 2023 at 17:38
  • 1
    @AugustoVasques: Questions start with individuals. Crap questions from an individual don't benefit the community. Users who ignore direction and correction from the community or platform on how to improve their question don't help. You can't build this ideal castle you're creating if your foundation is sinking sand.
    – Makoto
    Commented Apr 19, 2023 at 17:41
  • 5
    I think you are being too pessimistic. There is a cohort of "gimme teh codez" users who will never be able to participate fruitfully, and I agree that giving them ways to shine up their turds might be problematic. But there is also a sizable cohort of people who really try their best, but could use a little help the first few times.
    – tripleee
    Commented Apr 19, 2023 at 17:49
  • 4
    I think giving them three possibilities and forcing them to choose alleviates a lot of the potential for issues around picking the suggestion and assuming it's correct without checking.
    – Ryan M
    Commented Apr 19, 2023 at 17:56
  • 6
    this seems a bit silly, we aren't here to teach people how to write good titles or even good questions. we're here to create a useful knowledgebase of questions and answers. Yes, writing good questions, titles, and answers are all important aspects of this project, however, partially automating one part of it in a way that is bound to improve quality is hardly a step backwards.
    – Kevin B
    Commented Apr 19, 2023 at 18:17
  • 6
    The point that everyone seems to be missing here, and what Makoto seems to be actually be worried about, is that I do not trust the AI to generate actually meaningful titles 100% of the time. And not every bad title is caused by an inexperienced user: many times the title is bad because the user does not care at all. So, in this scenario, do you expect the user to avoid picking out a nonsense? Because if that happens, congrats: you just made the question harder to notice to editor by hiding its issue under an grammatically correct title that actually makes no sense to the actual question
    – SPArcheon
    Commented Apr 20, 2023 at 12:32
  • 6
    @SPArcheon: Yes, this is the point I'm making. If it were the case that these were suggestions and not something someone could just copy-paste into the title and call it good, then okay, I'd be cautiously optimistic. But if someone don't know how to title their question, how can they validate that their title makes sense at all?
    – Makoto
    Commented Apr 20, 2023 at 15:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .