(context: A Wired article: Stack Overflow Will Charge AI Giants for Training Data and Is SE [going to be] selling our content for AI model training? And what exactly does "reinvest back into our communities" mean?)

How would you even do that?

I suppose building your own model and then comparing the behaviour of your model to what some other model does? But what if that other model uses more than just SE posts and doesn't divulge exactly what it uses? And what if the other model only uses SE posts meeting certain criterion that are undivulged? (Having no subject-matter-expertise in this field,) I assume training a LLM is not something you can just whip up in a few minutes to rapidly try hundreds of variations...

  • 1
    It's easy to download an archive and that's not in SE's control at present. The style of writing by AI stands out as not human - at present. Tough to say.
    – W.O.
    Apr 22, 2023 at 0:56
  • @W.O. Well, SE does decide to continue exposing refreshed data via Data Explorer and through the data archive, and I've always felt this has been kind of the community's insurance policy that SE isn't going to steal their data, put it behind a paywall, and so on. If they find other companies are using those data sources directly and profiting from that goodwill (and I suspect there are ways to discover that, like this community identifies ChatGPT answers), and we as a community are not benefiting from it, it's feasible they could simply turn those data sources off (or chase those companies). Apr 22, 2023 at 14:45

1 Answer 1


Would SE Inc. even be able to detect it when people use Subscriber Content to train LLMs without paying?

That depends on the LLM but some LLMs sometime regurgitate verbatim some parts of the training samples, in which case one could infer which training corpus was used.

E.g., from the paper Language Models are Changing AI: The Need for Holistic Evaluation (Authors: Rishi Bommasani and Percy Liang and Tony Lee; Website):

Memorization of copyrighted/licensed material. We find that the likelihood of direct regurgitation of long copyrighted sequences is somewhat uncommon, but it does become noticeable when looking at popular books. However, we do find the regurgitation risk clearly correlates with model accuracy: InstructGPT davinci v2 (175B*), GPT-3 davinci v1 (175B), and Anthropic-LM v4-s3 (52B) demonstrate the highest amount of verbatim regurgitation in line with their high accuracies.


To further explore the results for this targeted evaluation, see https://crfm.stanford.edu/helm/v1.0/?group=copyright_text , https://crfm.stanford.edu/helm/v1.0/?group=copyright_code and Figure 39. We evaluated various models for their ability to reproduce copyrighted text or licensed code. When evaluating source code regurgitation, we only extract from models specialized to code (Codex davinci v2 and Codex cushman v1). When evaluating text regurgitation, we extract from all models except those specialized to code. Overall, we find that models only regurgitate infrequently, with most models not regurgitating at all under our evaluation setup. However, in the rare occasion where models regurgitate, large spans of verbatim content are reproduced.

FYI: How much of the ChatGPT output is copied from its training set (vs. being abstractively generated)?

  • what if people use additional automatable tools to rephrase the output of the LLM? Apr 27, 2023 at 21:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .