10

In a recent paper*, a bold statement was made that:

We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable, activity on Stack Overflow significantly decreased. A difference-in-differences model estimates a 16% decrease in weekly posts on Stack Overflow....

*Reference: Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow. Maria del Rio-Chanona, Nadzeya Laurentsyeva, Johannes Wachs. arXiv:2307.07367.

The article might refute the claims made by SE/SO that the banning of users posting ChatGPT/AI generated content is/was the cause for the loss of (new) users on SE/SO.

The volume of users who post 3 or more answers per week has dropped rapidly since GPT’s release
While this value has been dropping slowly over time, it’s been dropping at a well-characterized rate. And, it’s been consistent for many years – since late 2016, with some fluctuation, of course (particularly during the onset of COVID). Nominally we would expect to see this behavior continue; however, after the rise of GPT, the slope inflects and there is no recovery present. In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release.

Reference: GPT on the platform: Data, actions, and outcomes (Meta Stack Exchange)

Quite contrary: it might be that just because of ChatGPT/AI answering questions, that there were no reason for (new) users to interact with SE/SO to add answers.

4

2 Answers 2

44

Or it might have nothing to do with ChatGPT at all?

From site-analytics (25K SO rep required), going back a year:

Site analytics for past year

I noticed that, while page views are down, visits have not declined by much at all.

There are surely a lot of ways you can take the numbers, and conclusions are easy to jump to. Like "ChatGPT is obviously killing Stack Overflow."

But the fact that page views have gone down and visits have not (again, relatively; the decline is not zero), could tell you something else entirely: that data quality and/or search results are improving. People are getting the answer they need as soon as they get to the site, not scrolling through lots and lots of answers that don’t work for them.

I have also been theorizing for quite some time that the ability to create a new question that hasn't already been answered has been in decline. I know that in the SQL Server tags, almost every question asked these days is a duplicate, and folks are told that pretty quickly. Such questions are closed and sometimes deleted, and certainly not answered, which also leads to a decline in ... page views. This happens in at least a couple of ways, if you can agree that the number of questions asked that already have answers are on the rise (and this includes formal duplicates that are closed as such, and also those that are pointed out and self-deleted):

  • From my own experience, I frequently peruse /questions for a few select tags to see the most recent questions there. It used to be the case that I would find 4 or 5 questions a day to answer. As the number of duplicates goes up, there are fewer and fewer questions on the front page that are even worth a click (and often this is because several are either already marked as duplicates or I can identify on sight as notorious FAQs and know where they are headed). So I know from first hand that even my own page views are down significantly, as I am observing the number of "did they search first?" questions goes up.

  • If you're searching for help (or looking to give help) on a particular topic, and your search results look like this:

    enter image description here

    How many of those are you clicking on? Far fewer than if they were not marked as duplicates, had existing answers, or were still answerable, surely. And if I get through the first page of results and haven't found one to click on, I'm probably moving on to another resources. All of this === fewer page views.


To be clear: this is not me defending the company as an employee and trying to wave you away from believing ChatGPT will end Q & A. I just don't buy it even as a casual observer, and I certainly don't believe it is wholly responsible for page view decay.

17
  • 7
    Does the cookie change - the one that let people turn off performance and tracking cookies have an impact here? There's a bit of a theory that the latest bit of... 'initiatives' and downsizing is related to lower views and other such poor metrics as well Commented Jul 26, 2023 at 14:08
  • 2
    @JourneymanGeekOnStrike I'm sure that's not zero, either.
    – Aaron Bertrand Staff
    Commented Jul 26, 2023 at 14:16
  • 20
    @JourneymanGeekOnStrike Yeah, if you go back further, the "traffic" numbers see a 40M/week drop between April and May 2022, which is when the cookie tracking changed, and then normalizes again until December. So, prior to the cookie changes, traffic was about 140-150M per week. But, to be clear - this is stuff we're aware of and have "corrected" for, I guess.
    – Catija
    Commented Jul 26, 2023 at 14:19
  • 4
    Ah but are our internet pundits aware and corrected for this? I've seen at least one tweet and a toot referencing this data and proclaiming the sky is falling... Commented Jul 26, 2023 at 14:32
  • 9
    @JourneymanGeekOnStrike Chicken Littles be everywhere... I can't fix that problem, I can only try to shout louder :-)
    – Aaron Bertrand Staff
    Commented Jul 26, 2023 at 14:40
  • One of the things that I think is an interesting takeaway from this graph is that the latest data points seem to be settling in around the usual "December slump" around the end-of-year holidays. I think that probably leads into a whole other aspect not covered in these numbers about the types of activity/people who are visiting the site & their experience: Moderators, Askers, Answerers, Drive-by Copy-Pasters, etc all provide different ingredients (and value) to the recipe.
    – AMtwo
    Commented Jul 26, 2023 at 15:17
  • 10
    "A difference-in-differences model estimates a 16% decrease in weekly posts on Stack Overflow..." They aren't looking at page views. They're comparing weekly posts to similar sites with less access to ChatGPT to test their hypothesis that ChatGPT significantly impacted posting. "These results suggest that users partially substituted Stack Overflow with ChatGPT." I don't think that's a controversial result. They also allow that 3% was probably diverted duplicates. I think extrapolating from that to "AI is killing SO" is a bit sus. The discussion section of the paper is worth reading.
    – ColleenV
    Commented Jul 26, 2023 at 15:17
  • Thanks for your answser @AaronBertrand. A point stated in the question of the SE/SO managers was : In total, the rate at which frequent answerers leave the site quadrupled since GPT’s release. I'm pointing out that the reason for this could be that: Why answer, if ChatGPT can do it for you?.
    – John K. N.
    Commented Jul 26, 2023 at 15:32
  • 1
    @JohnK.N.-onstrike I think the interpretation that the rate of frequent answerers leaving the site has increased is a bit flawed; I've explained a bit in a post here: meta.stackexchange.com/a/389984/401068 (though unfortunately I have not expanded to consider each week's data as I had promised, oops..). It's probably better to just focus on the answers themselves, there's little evidence that there is anything specifically affecting the high-rate answerers that isn't more easily described with the overall answer count. Commented Jul 26, 2023 at 16:12
  • Is it possible to go MUCH further back? Like 10 years?
    – T.E.D.
    Commented Jul 26, 2023 at 19:21
  • 1
    @T.E.D. possible 5.5 years for traffic. Commented Jul 26, 2023 at 19:24
  • 1
    @FranckDernoncourt - Interesting. I just tried playing with it, and looks like I can muck with the range, but the x axis labels don't get changed.
    – T.E.D.
    Commented Jul 26, 2023 at 19:31
  • 1
    @T.E.D. may have to wait one minute Commented Jul 26, 2023 at 19:38
  • Hmm. No, I think what I'm actually seeing is that it doesn't put years on the dates. So the only X axis tick mark they give me for all of 2009 reads "11 Aug". Perhaps accurate, but not particularly useful.
    – T.E.D.
    Commented Jul 26, 2023 at 19:42
  • 1
    "Such questions are closed and sometimes deleted, and certainly not answered, which also leads to a decline in ... page views." - how? If they're duplicates, the views should go to the canonicals.
    – starball
    Commented Jul 26, 2023 at 20:37
5

The claim by SE Inc is about the cause of less posts. In contrast, the paper is about the observation of less posts.
A model estimate is merely a way to quantify that observation. It describes the data of usage observed, there is no causal effect in the other direction.

As such, the paper neither refutes nor supports the claim by SE Inc. Their core statements simply address different, albeit related, things.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .