Correlation between AI generated content and the demise of SE/SO

Question

In a recent paper*, a bold statement was made that:

We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable, activity on Stack Overflow significantly decreased. A difference-in-differences model estimates a 16% decrease in weekly posts on Stack Overflow....

^{*Reference: Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow. Maria del Rio-Chanona, Nadzeya Laurentsyeva, Johannes Wachs. arXiv:2307.07367.}

The article might refute the claims made by SE/SO that the banning of users posting ChatGPT/AI generated content is/was the cause for the loss of (new) users on SE/SO.

The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release
While this value has been dropping slowly over time, it’s been dropping at a well-characterized rate. And, it’s been consistent for many years – since late 2016, with some fluctuation, of course (particularly during the onset of COVID). Nominally we would expect to see this behavior continue; however, after the rise of GPT, the slope inflects and there is no recovery present. In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

^{Reference: GPT on the platform: Data, actions, and outcomes (Meta Stack Exchange)}

Quite contrary: it might be that just because of ChatGPT/AI answering questions, that there were no reason for (new) users to interact with SE/SO to add answers.

Answer 1

Or it might have nothing to do with ChatGPT at all?

From site-analytics (25K SO rep required), going back a year:

I noticed that, while page views are down, visits have not declined by much at all.

There are surely a lot of ways you can take the numbers, and conclusions are easy to jump to. Like "ChatGPT is obviously killing Stack Overflow."

But the fact that page views have gone down and visits have not (again, relatively; the decline is not zero), could tell you something else entirely: that data quality and/or search results are improving. People are getting the answer they need as soon as they get to the site, not scrolling through lots and lots of answers that don’t work for them.

I have also been theorizing for quite some time that the ability to create a new question that hasn't already been answered has been in decline. I know that in the SQL Server tags, almost every question asked these days is a duplicate, and folks are told that pretty quickly. Such questions are closed and sometimes deleted, and certainly not answered, which also leads to a decline in ... page views. This happens in at least a couple of ways, if you can agree that the number of questions asked that already have answers are on the rise (and this includes formal duplicates that are closed as such, and also those that are pointed out and self-deleted):

• From my own experience, I frequently peruse /questions for a few select tags to see the most recent questions there. It used to be the case that I would find 4 or 5 questions a day to answer. As the number of duplicates goes up, there are fewer and fewer questions on the front page that are even worth a click (and often this is because several are either already marked as duplicates or I can identify on sight as notorious FAQs and know where they are headed). So I know from first hand that even my own page views are down significantly, as I am observing the number of "did they search first?" questions goes up.

• If you're searching for help (or looking to give help) on a particular topic, and your search results look like this:

  

  How many of those are you clicking on? Far fewer than if they were not marked as duplicates, had existing answers, or were still answerable, surely. And if I get through the first page of results and haven't found one to click on, I'm probably moving on to another resources. All of this === fewer page views.

---

To be clear: this is not me defending the company as an employee and trying to wave you away from believing ChatGPT will end Q & A. I just don't buy it even as a casual observer, and I certainly don't believe it is wholly responsible for page view decay.

Answer 2

The claim by SE Inc is about the cause of less posts. In contrast, the paper is about the observation of less posts.
A model estimate is merely a way to quantify that observation. It describes the data of usage observed, there is no causal effect in the other direction.

As such, the paper neither refutes nor supports the claim by SE Inc. Their core statements simply address different, albeit related, things.