Our robots.txt file is changing to allow only CloudFlare's verified bots

Question

I’m elo, Stack Exchange’s Director of Reliability Engineering. I’ve been employed at Stack Exchange for just over three years - but this is my first time posting here on Meta!

On December 10th, our robots.txt file will change. This change is part of our efforts to prevent unauthorized automated access to the network.

Currently, our robots.txt file is used to direct search engine crawlers to parts of the site that should or should not be indexed. It is also currently used to disallow specific crawlers that we find to misuse the content of the network in some way. We are switching to a model where we allow only those crawlers that are trusted to use the content on the network properly.

Implementation details

Currently, if you visit our robots.txt file, you will see a long file detailing which parts of the site should and should not be indexed, as well as which crawlers are expressly prohibited from crawling the site.

After this change, any trusted crawler will continue to see the old copy of robots.txt. Any unauthorized crawler will see the following copy instead:

User-agent: *
Disallow: /

To determine which crawlers are trusted and which are not, we are relying on CloudFlare’s list of verified bots. If you are running a crawler and want access to crawling Stack Exchange’s servers, we strongly recommend you follow CloudFlare’s process for adding bots to their list of verified bots.

Why are we making this change?

Beginning with the release of GPT-3 in November 2022, we noticed a significant amount of bulk content scraping activity taking place across the network. Over the last two years, the rate and volume of content scraping activity has ramped up dramatically; over the last year, external scraping grew by one million requests per day.

In the last year, Stack has seen numerous data points that suggest LLM providers have escalated their methods for procuring community content for commercial use.

When LLM providers, or other data sourcing companies, scrape content from Stack Exchange, they often train their models on Stack Exchange content without returning any portion of their benefits to the communities that provided the knowledge. Typically, they do not use that content in a way that enables attribution, either.

As a result of these misuses of content hosted on Stack Exchange, we laid out our vision regarding the changing state of the internet, and how it informs the way we operate.

A key component of our plan necessitates taking protective measures to remove access to Stack Exchange data from LLM providers that are misusing it to train foundation models without attribution. Changing our robots.txt file is the next logical step to express our belief that unknown entities should not be scraping the site.

Some reassurances: what this change is not

We recognize that this change may look very similar to a series of farther-reaching changes certain other services around the Internet have recently made. For that reason, our goal in this section is to explicitly reassure you about four sensitive topics.

First, this change does not represent an attempt to sell search indexing access to third parties. Stack Exchange’s mission is to provide open access to the repository of human knowledge stored here, and we do not plan to restrict access to users. It is our firm belief that restricting who can search for content on the network would not benefit either the users of the network or the company. The primary way users across the Internet access Stack Exchange is via search engines, so it stands to reason that inhibiting search engine functionality would not be beneficial.

Second, based on our analytics, only a very small proportion of users can expect to see any impact to their search results. For a user to be affected by this change, that user would have to use a search engine that builds their own search engine index (rather than relying on an extant well-recognized index), and that search engine company’s crawler must be listed on CloudFlare’s list of trusted bots. We will, however, be monitoring the metrics we capture about rate-limits and blocks to ensure no outsize change is detected.

Third, this change is not being made to aid in sales at the company. It is instead firmly rooted in our belief that modern web crawling and scraping require transparency and accountability, particularly in light of the misuse of content for training generative AI foundation models.

Finally, this change will not make it more challenging for users to export or analyze Stack Exchange data in bulk. Neither personal data export nor via per-site data dumps will be affected, nor will there be any impact to community-driven tools as a result of this change.

Further questions

We understand that this change is going to raise questions. We encourage you to leave any questions you have as answers below. We respectfully ask that you limit your questions to one per answer in order to facilitate smooth conversation.

Answer 1

The robots.txt file is really just a request to crawlers pointing them where to crawl.

Quoting this answer from Software Engineering:

The Robot Exclusion Standard is purely advisory, it's completely up to you if you follow it or not, and if you aren't doing something nasty chances are that nothing will happen if you choose to ignore it.

Reputable crawlers do follow the instructions in robots.txt, because they are reputable. Disreputable crawlers will simply ignore it.

It sounds like the Company's goal is to stop disreputable scrapers, which are already abusing the content by failing to follow license terms. However, I would expect disreputable scrapers that already ignore license terms to simply ignore the instructions in robots.txt.

I know from my experience working for you in Reliability Engineering that problematic scrapers are traditionally python scripts, and not full-featured bots. This adds to my expectation that the problematic scrapers are already, or will respond with ignoring the robots.txt.

In the spirit of transparency & education of the community, can the Company share details on the research into what portion of crawler traffic they expect to deflect with this change to robots.txt?

Answer 2

I agree with AMtwo - the use of robots.txt is really only applicable to well-behaved crawlers. It does nothing against bad crawlers or other bad actors scraping and stealing SE content.

I also find the argument about LLM providers to be very weak. In the United States, it's still an open legal question of whether and when training a model is fair use. There is ongoing litigation that could begin to resolve this issue, but I would also expect many appeals and a rather drawn-out process. The Creative Commons (who authored and maintained the licenses used for the SE Network content) maintains a very different stance than the company on the training of LLMs and other AI models (1, 2).

I share the company's concerns regarding the output of AI models, especially when those models reproduce or base their output significantly upon other people's work. There is a need to cite or attribute, depending on the nature of the use. However, this is a different problem than training the model.

Since you are blocking unknown entities, I'm curious as to what the impact on the network truly is. Are these bad actor crawlers placing such a strain on the network - either technically or financially because of the need for scaling - that they must be blocked?

Unless there are technical or financial reasons to block bad actors, I find it disheartening that the company is unilaterally making the decision to block access to our content for training purposes. It's one thing if it's the will of the community, but, to the best of my knowledge, that discussion has never been had with the contributors.

Given the fact that the primary rationale is still an open question and the answer of experts tends to disagree with the company's analysis and the fact that this does little for actual bad actors, I'd like to better understand the investment of time and resources into this over any of the other things that the community is asking for.