In a recent statistics post, the company reported on a large false positive detection rate of a specific GPT content detection tool when applied to old (pre 2023) content. Later in that post they introduced a measure they think represents the amount of copy-pasted AI generated content, the ratio of large to small draft saves, and emphasized that this measure cannot be used to reliably inform about the state of a single post but can be used to estimate the average rate of such content.
That made me thinking if there are maybe more measures, that can be used to estimate the average rate of GPT content since November 2022?
For possible candidates one would basically search for anything that changed significantly around that time and cannot be explained otherwise.
I came up with one example: maybe the average length of a copy-pasted GPT answer is different from the average length of a human written answer. I checked this by calculating the average length of an answer for the last three years on Stack Overflow and saw that indeed starting with November 2022 the answer length of SO increased and is now at roughly 15% longer than it before while before it was relatively stable. The fluctuations in the data are much smaller than the observed effect.
However, I'm not completely satisfied. I could not calculate the average length of deleted answers (in PostsWithDeleted field Body is empty) so this effect would only be for still existing answers and I wanted to calculate percentiles (25%, 50%, 75%) but that proved slightly too difficult for me in SQL. Also when looking at other StackExchanges like SuperUser or Mathematics, I did not see such an effect. It could simply be that in 2023 experts on Stack Overflow decided to provide more thorough answers.
Are there maybe other unexplained changes around that time (number of edits, comments, timings between answers by same author, occurrence of certain phrases in post bodies, ...)?
Maybe the company can help there too, because for deleted posts some data is removed in SEDE but the deleted posts are presumably rich in GPT content and would be nice to analyze too.