28

In a recent statistics post, the company reported on some measures they regard as relevant in the current AI usage debate.

One is the ratio of large draft saves to small draft saves, which is supposed to correlate with the amount of copy-pasted AI content. It peaked shortly after the publication of ChatGPT and then fell back to levels of before that period.

The other one is the number of answerers who post at least three answers per week, seen as a pool of valuable experts. This number dwindled quite fast and remained declining with a large portion of the decline attributed to suspensions. The high number of suspensions were related to implementation of the AI content ban.

Both measures were mainly calculated for Stack Overflow.

At the end of May the company announced internally (exact details are still not known) some kind of ban on the suspension of AI content posters and a large fraction of moderators subsequently went on a strike. Effectively one can assume that since the begin of June it's possible to post AI generated content without getting penalized.

How have these two measures developed since then?

Ideally I'd like weekly data points for the last say 52 weeks and with an update in 2-3 months in order to compare the development.

If there was for example a rebound visible to early ChatGPT time levels, my belief that they are connected with AI content creation would be strengthened. If not, the situation might be more difficult to interpret, but nevertheless it would be interesting to know in my eyes and the data is there, so it's cheap to produce some graphs about it. And with thousands of posts on Stack Overflow every day, the noise influence on the curves should already be limited.

Improve this question
12
  • 2
    Since it is related and probably not worth a separate question: How badly was and is this metric messed up by whatever is increasingly wrong with Googles gatekeeping and by hitting the rate limit in connection with the <del>keylogger</del> small draft saves?
    – anx
    Jun 27 at 1:38
  • Just a note on "Effectively one can assume that since the begin of June it's possible to post AI generated content without getting penalized." - That's… only somewhat true temporarily. If someone is posting many ChatGPT answers and gives an excuse of "the mods went on strike", there’s a very solid chance they'll get suspended post-strike. So to anyone considering trying to post AI-generated answers: please don’t.
    – cocomac
    Jun 27 at 7:02
  • Re "since June": We are still in June (2023). Do you mean a particular date in June 2023?
    – This_is_NOT_a_forum
    Jun 27 at 8:00
  • @This_is_NOT_a_forum thanks, I specified the date in the title.
    – NoDataDumpNoContribution
    Jun 27 at 9:28
  • 2
    @cocomac Nobody knows how the strike will end. You seem to assume a specific outcome but couldn't it also be that the current company policy actually remains as is? Actually I didn't think much about possible future developments.
    – NoDataDumpNoContribution
    Jun 27 at 9:30
  • 2
    @cocomac Out of curiosity, if I'm a new user where do I see in the posting guidelines or help center that posting content generated in part by AI will result in suspension? Most of the new users to ELL have no clue what ELL's meta is or that they should be looking for site policies there. From what I've seen, the ban on AI has only been posted in the site metas. There's nothing in meta.stackexchange.com/help/behavior that says "Your posts should come from your brain and not be generated by AI".
    – ColleenV
    Jun 27 at 12:25
  • 2
    @ColleenV Stack Overflow has a page on the ban in its help center. Not sure about other sites, though the meta thread often is featured/hot/… .
    – MisterMiyagi
    Jun 27 at 18:28
  • 2
    @MisterMiyagi Thanks! That's something at least. When I go to post an answer when I'm logged out, the only guidance I'm directed to is stackoverflow.com/help/how-to-answer and there's no mention of looking elsewhere in the help for policies. That seems a bit unfair to new users who know nothing about SO and how it works. I hate to dredge up the new user onboarding stuff again, but if y'all are suspending people without warnings for posting ChatGPT stuff, and every site has a different policy, and that policy isn't obvious when posting an answer, that's not ideal.
    – ColleenV
    Jun 27 at 20:04
  • 4
    @ColleenV They declined to give any site other than SO that Help Center page (which by the way has definitely not stopped AI on SO). Mods have asked for more, like a banner on the answer box, but I've not seen any indication from staff that such a feature (or similar) will be implemented.
    – Laurel
    Jun 27 at 20:24
  • 2
    @Laurel Thanks. I figured it was something like that. Once again, volunteers get thrown under the bus for trying to solve a problem without proper support from the company.
    – ColleenV
    Jun 27 at 21:34
  • 8
    It's possible to query the 3+ answerers on SEDE. The last two weeks are up from the low point, but it's not yet possible to determine if this is noise or somehow related to the moderator strike. I also am curious about the drafts since I'm somewhat skeptical they identify typical ChatGPT usage.
    – Jon 'links in bio' Ericson
    Jun 27 at 23:06
  • 3
    @ColleenV On talking with other mods, I learned that mods pushed for clearer warnings (like one on the answer box like I mentioned) as part of strike negotiations. Apparently this point has been agreed to, but they're making it "a full feature" aka overcomplicating it when we really need it now. (So the previous comment wasn't quite accurate. Ah, the hazards of writing comments purely from memory. In all fairness I'm not sure where/if this was publicly shared on SE.)
    – Laurel
    Jun 28 at 0:42

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .