Announcing a change to the data-dump process

Question

UPDATE August 29, 2024

This new post has updates related to various bug reports and feature requests.

UPDATE August 14, 2024

Over the past week we’ve scaled up the availability of the new page and process. Thank you to everyone who tested it out. Effective today, this is now available to all users.

UPDATE July 26, 2024

Thank you for your patience as we worked to get clarity around a few issues. We have two updates.

First, our product team has regretfully informed us that it will not be possible, within the existing time constraints, to bundle the dumps for a site and its meta together. So if you want to get the download for a site and its meta, you will need to do the request on both sites. We know the initial announcement implied that the two files would be bundled together, and we’re sorry that won’t be the case right now. That bundling was initially assumed to be a low lift, but it turned out not to be, so was taken out of scope for this release.

The second update we have is regarding the “Click to agree” language. Many users have found that language to be unnecessarily bureaucratic and difficult to parse and understand, within the context of a CC BY-SA 3.0 license. Philippe committed to asking our lawyers to take a look, and they did.

Both the product and legal teams have signed off on this modified language:

"I understand that this file is being provided to me for my own use and for projects that do not include training a large language model (LLM), and that should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump."

This makes it easier to understand the agreement they’re making by downloading the file and, most importantly, clarifies that there is a separate path for someone who would like to use the data to train an LLM (using the “contact us” link below the heading). The “Become a partner” form allows someone to specify interest in non-commercial LLM-training use as well as commercial use. This updated image of the data dump access page (below) reflects the new language.

We’ll update this post again in the coming weeks when the data dump access page page is available for use.

---

Original Announcement

12 July, 2024

Today, we are announcing some changes to the data dump process. I’m going to start with an important statement: this is primarily only a change in location for where the data dump is accessed. Moving forward, we’ll be providing the data dump from a section of the site user profile on a Stack Exchange profile.

There are a number of reasons for this: first, this is an attempt to put commercial pressure on LLM manufacturers to join us and our existing partners in the “socially responsible AI” usage that we’re advocating for - to get them to give back to the communities whose data they consume.

Second, we want to help make the process of accessing data dumps quicker and more efficient. While the Internet Archive has been a great partner to us, as you may know, both internally and externally, people have encountered challenges with uploading and downloading the dumps with any reasonable speed.

Lastly with the heightened ease of receiving a dump, we’re curious to see what community members will build with this information.

I want to emphasize a few things:

1. You may download the dumps, free of charge, as you always have, just in a different and we hope more convenient location. The CC BY-SA license is unchanged.

2. The dump file will be provided in an “instant” format - we will generate a URL on the backend for the data from the site to be downloaded.

3. We are requesting that if you intend to use the dump for a commercial purpose, you consider joining the socially responsible AI movement and giving back to the community. For commercial purposes, there will be a link on the form to route you to the Stack Overflow team to discuss more information and next steps.

4. We are requiring that all partners in socially responsible AI comply with the CC BY-SA attribution requirements, attributing content to the community members who contributed it. All reuses of the data require attribution and it’s a part of the license.

Data dump access

You can access this site's data for non-commercial use. For commercial purposes, please contact us.

---

A new data dump is available every three months. Learn more about the data dump process.

(checkbox) I agree that I will use this file for non-commercial use. I will not use it for any other purpose, and I will not transfer it to others without permission from Stack Overflow. I certify that I am not downloading this file on behalf of my employer, for use in a for-profit enterprise. I have read and agree to the Terms of Service and the Acceptable Use Policy, and have read and understand Stack Overflow's privacy notice.

(button) Download data

(Above: The proposed location of the data dump access within the account settings; please note that some of the placement/layout/language is subject to change.)

At the same time, we are also reviewing our stance on data scrapers who pull down the site, including those who then use the data to impersonate Stack Overflow. We are adding additional checks for bots, and are hardening that infrastructure. This has been a longstanding request from the community, and we’re pleased to be able to resource this change. More details to come on this one.

Unfortunately, this new process will not be ready until mid-August. Because this is an important change, we want to use it for the July data dump release, which means that we will miss our previous commitment date of July 31 for the data dump. If you have an urgent and critical need for an updated version of the data, please contact us using the contact form, and we will see if there’s a way to accommodate you on the original schedule. The meta post will be updated once the new data dump process is available.

---

FAQ

Why are you updating the data dump process?

We are attempting to protect the long-term viability of the Stack Exchange network and to ensure that it is a comprehensive and well-attributed resource for generations to come.

We want to modernize the data dumps process and make it more efficient for users in the community. By adding the data dump process within the Stack Exchange site profile, users will be able to quickly access and track the quarter’s data dump.

Simultaneously, we know that companies have scraped or otherwise ingested Stack Overflow and Stack Exchange data to train models without proper attribution — models that they are monetizing or using for commercial purposes. We know this is happening because the companies themselves, or independent researchers, have disclosed this information. Assuring long-term viability of the community requires resources, and the commercialization process exists to assure that those resources flow back into this community. This also allows us to advocate on behalf of the contributing community members for appropriate attribution.

It’s important to note that this is not about preventing the everyday person from accessing the data dumps, nor is it about putting up a wall that is impervious to workarounds (which would be futile). This is part of laying out the pathway for companies to go through the proper channels and be above board in how they access the data. These companies want to be doing it the “right” way. In the rapidly changing legal and ethical landscape, being compliant and having a defined partnership for data access is important to both their brand and their bottom line.

This change should have little practical impact on the everyday person accessing the data dumps. It is ultimately a change in service of a very important goal. This is a trade-off in response to a rapidly changing world - a world in which we are now dealing with things like LLMs, which were not even a consideration when the CC BY-SA license was originally adopted.

How do users access data dump files?

To get the data file for a Stack Exchange site, there will be a location within the settings page on each Stack Exchange site. Users will be provided with the link to the data file, and can also continue to access the data through the Stack Exchange API or Data Explorer.

Please note that users will only be provided the data for the specific Stack Exchange site and its Meta site for the corresponding Stack Exchange site profile. If you wish to access the data for additional Stack Exchange sites, you will need to go through the same process for each site.

How is a request for commercial use made?

Commercial use requests can be made by filling out the “Become a partner” form on this page. That will be passed to our team who is responsible for working with partners.

Will Stack Overflow be uploading the data dump to archive.org?

Stack Overflow is no longer uploading the data dump to archive.org. We initially began to host the data dump on archive.org when our prior host went out of business. Now, we have developed the in-house capacity to host these ourselves. We would really rather users do not upload the file to archive.org or similar data pile sites. Assuring the viability of the Network takes resources, and companies that profit off the back of the work of this community should feel an obligation (“socially responsible AI”) to give back to the communities whose work they use to create the models that they are marketing commercially. Our hope is that because the process for individuals to request the dumps is lightweight and quick, you won’t feel the need to undermine these efforts to encourage commercial re-users to contribute back.

Can a user give the file to their employer for them to train a language model on?

When organizations are able to skip out on their obligations to contribute back, the whole internet suffers. Without Stack Overflow as a resource, many of the world’s millions of technologists would not be able to find the answers to their programming questions.  And we know that these organizations are not contributing voluntarily - so putting this light commercial pressure on them is the best method to encourage socially conscious behavior.  It’s important to say that when you breach the agreement that you make when downloading the dumps file, we do have the option to decline to provide you with future versions of the data dumps.  But we really don’t want to have to do that.

A user has an unusual need for the full data dump file. How can they get that fulfilled?

We're scoping and building a process to allow for this, but it will not be in the initial release of the feature. Look for this in v2. For now, if you have a critical need for the full file, please write to us using the contact form and we will try to assist.

Answer 1

On July 26, Stack Overflow updated the post to include different modifications to the CC BY-SA license than the prior version. This remains a breach of the license terms, as the CC BY-SA license does not allow additional restrictions, as previously outlined below in this answer and in other answers.

---

There is a lot going on here... I'll come out and say it that after first reading, it feels like a great big distraction because Senior Leadership is treating the data dump like a boogeyman, when that's wholly unfounded.

That said, let me try to organize my thoughts some.

Missing the July Dump deadline.

Just over a year ago when I was still staff at the company, I was personally in the unenviable position of having been instructed by the Stack Overflow CEO to disable the Data Dump, and to not re-enable it because he wanted to end the dump. That decision ultimately snowballed until Stack Overflow made commitments to continue the data dump quarterly. Data Superstar Aaron ultimately made some improvements and there was a shift made to the delivery schedule, to make it align better with quarterly boundaries. This is all excellent news for those of us who use the data dumps, and/or are proponents for equal data, and/or are defenders of the open data commitments made by and for the community.

Now, just one quarter after the company's most recent commitment to a schedule, it's shifting, again. For no reason. Apparently undoing the most recent schedule-shift by bumping (at least) a month.

WHY CAN'T THE DUMP BE POSTED TO ARCHIVE.ORG ONE MORE TIME?

There is no rational reason given as to why to delay the July dump. There are no blockers preventing the company from continuing the existing process until the new process is ready.

Stack Overflow has had plenty of time

Evidenced by the Data Dump's recent history, the company has had plenty of time to pursue changes. It certainly seems like you set an arbitrary deadline, missed it, and are going to now cause arbitrary delays because YOU did not prioritize the work well enough to meet your own deadline. I won't belabor why that's problematic.

If you want to build good will with the community, I'd suggest that the data dump promised to the community by July 31, 2024 still be delivered by that deadline, regardless of the state of the new process you want (but do not need) to use.

Stack Overflow will be violating the BY-SA license

Policing commercial use goes against the Creative Commons BY-SA license

The Creative Commons license is very clear (emphasis mine).

Share — copy and redistribute the material in any medium or format for any purpose, even commercially.

Ethical AI is a great talking point, but I'm not sure of the ethics behind preventing commercial use of something when it is legally, explicitly available for commercial use. The freedom to use it for commercial purposes obviously comes with the need to follow all license terms. The Creative Commons license explicitly says this:

The licensor cannot revoke these freedoms as long as you follow the license terms.

In essence, Stack Overflow is limited to ensuring that downstream users provide Attribution, and that the data continues to be shared under the same open license. Again, quoting from the BY-SA license:

No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

For folks reading along, this last quote from the license does NOT disallow requiring a username/password to limit access, but does prohibit any sort of DRM, "watermarking", or other means to limit use.

This license quote DOES prohibit the checkbox included in the data download mockup. By requiring a user to "agree that [they] will use this file for non-commercial use...and [they] will not transfer it without permission..." Stack Overflow is violating the license terms. Period.

A breach of the license terms results in automatic termination

From the Creative Commons FAQ:

All CC licenses are non-exclusive: creators and owners can enter into additional, different licensing arrangements for the same material at any time (often referred to as “dual-licensing” or “multi-licensing”). However, CC licenses are not revocable once granted unless there has been a breach, and even then the license is terminated only for the breaching licensee.

If Stack Overflow violates the CC BY-SA license, the users who created the content (ie, "us" not the Company) can terminate the license granted to the Company. This could be DEVASTATING to the community. Particularly on smaller SE sites, a small set of users forcefully revoking Stack Overflow's ability to use the data under CC BY-SA could set a site back years.

I would be excited to see the Company helping to enforce BY attribution and SA share-alike licensing by others on the internet. However, it is incredibly disheartening for individual contributors to see the Company being the potential breacher, rather than defender.

The data dump is licensed under CC BY-SA explicitly.

Quoting from the Stack Overflow Terms of Service:

From time to time, Stack Overflow may make available compilations of all the Subscriber Content on the public Network (the “Creative Commons Data Dump”). The Creative Commons Data Dump is licensed under the CC BY-SA license. By downloading the Creative Commons Data Dump, you agree to be bound by the terms of that license.

Users grant Stack Overflow the CC BY-SA license. Stack Overflow uses that license to "remix" the individual posts into the compiled data dump, and appropriately covers the entire derivative work under the same CC BY-SA license. By requiring any individual to agree not to use it for commercial purposes, Stack Overflow would be violating the "no additional restrictions" clause, and be subject to automatic revocation of the CC BY-SA license from the grantor (users, authors, content creators).

"We would really rather users do not upload the file to archive.org or similar data pile sites"

I can appreciate that the company might rather this not happen. But unfortunately, the CC BY-SA license means that the Company can't restrict this. Stack Overflow could try changing the actual format, and licensing that anthology format differently, and place restrictions on that new archive product, but the content itself will always be free to be uploaded to someplace like Archive.org.

"When organizations are able to skip out on their obligations to contribute back..."

Unfortunately, this is not an obligation that is covered by the CC BY-SA license. And even more unfortunately, adding this restriction on top of the BY-SA license would be a breach of that original license, and thus Stack Overflow would be the entity in legal hot water, not the downstream users that Stack Overflow is trying to police.

Let's assume Stack Overflow proceeds...

even in a unique format to circumvent CC BY-SA...

Because many of us are technologists and software developers, I can almost guarantee that someone will create a process that:

• downloads the Data Dump for individual, hobby use
• creates a process to ingest the data dump, and reformat as XML if necessary
• uploads the new data in a backwards-compatible format to archive.org

Stop Gaslighting me

It’s important to say that when you breach the agreement that you make when downloading the dumps file, we do have the option to decline to provide you with future versions of the data dumps. But we really don’t want to have to do that.

• User-generated content is licensed under CC BY-SA to the entire world.
• Stack Overflow compiles those CC BY-SA licensed creations into a data dump, and then licenses the product that is the data dump under the same license.
• The "Creative Commons Data Dump," being licensed under the CC BY-SA license, permits any use, including commercial, so long as derivative works include the S and the A--"Share Alike" (continued BY-SA licensing) and "Attribution".
• By attempting to enforce a non-commercial limitation on a BY-SA licensed creation, Stack Overflow is breaching the agreement with the community.

It’s important to say that when you breach the agreement that you make when creating the dumps file, we do have the option to revoke the license we granted when we posted content on the site. But we really don’t want to have to do that.

Dual licensing

Nearly everything I said above has a big caveat... At the time of posting, users granted the entire world a CC BY-SA license, and additionally granted a second license to Stack Overflow (emphasis mine):

You agree that any and all content, including without limitation any and all text, graphics, logos, tools, photographs, images, illustrations, software or source code, audio and video, animations, and product feedback (collectively, “Content”) that you provide to the public Network (collectively, “Subscriber Content”), is perpetually and irrevocably licensed to Stack Overflow on a worldwide, royalty-free, non-exclusive basis pursuant to Creative Commons licensing....and you grant Stack Overflow the perpetual and irrevocable right and license to access, use, process, copy, distribute, export, display and to commercially exploit such Subscriber Content, even if such Subscriber Content has been contributed and subsequently removed by you

Stack Overflow could use their perpetual, irrevocable license to do anything they want with the data. Stack Overflow doesn't have to release the "Creative Commons Data Dump" under the CC BY-SA license -- except that it does. Stack Overflow could change the TOS to stop licensing the Data Dump under CC BY-SA, and then add any restrictions they like. This would essentially be a proprietary "box" filled with freely reusable data--folks would be bound by the Company's terms for the "box", but they could take the CC BY-SA contents out of the box, throw away the box, and use the contents of the box in any way that meets the terms of the CC BY-SA license (including putting that data into a CC BY-SA box).

However, at the end of the day, everyone in the world can use the post content however they like, so long as they continue to follow the CC BY-SA license restrictions, regardless of how they access that data.

My promise to Stack Overflow

I intend to vigorously defend my rights as a content creator on the Stack Exchange Network. I will ensure that my content (which I licensed to the entire world under CC BY-SA) continues to be used properly according to the license terms. If/when someone breaches the license terms, I will revoke that license and defend my content's copyright.

Answer 2

How do you plan to enforce "I agree that I will use this file for non-commercial use. I will not use it for any other purpose, and I will not transfer it to others without permission from Stack Overflow." when the CC BY-SA license explicitly forbids adding downstream restrictions?

• No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.

Answer 3

to get the data file for a Stack Exchange site, there will be a location within the settings page on each Stack Exchange site. Users will be provided with the link to the data file, and can also continue to access the data through the Stack Exchange API or Data Explorer.

At best, this is extremely inconvenient; at worst, it guarantees no one will ever again have a consistent "dump".

I'm going to guess: no one involved in making this decision has ever downloaded and worked with the full data dump. It's already slow and fairly inconvenient; the one bright spot is that a decent torrent client lets you start it and do other stuff while waiting. Best-case, you devote a fast enough pipe to this that the hundreds of extra clicks necessary are rewarded with shorter turnaround... But somehow, I doubt it. So instead, we get a tedious click game.

Oh well. I can script it. There'll probably be more failed or interrupted downloads, but I can script around that too. Annoying, but feasible.

We are adding additional checks for bots, and are hardening that infrastructure.

Oh... You thought of that too. So this is literally, "Let's make using the data our users have contributed over the years as painful as possible" then.

Thanks for all the lies.

Answer 4

we want to help make the process of accessing data dumps quicker and more efficient

Umm. OK, I won't laugh out loud, but instead give some simple facts.

This is the process today:

1. Anyone can download all 180+ data dumps (one of each site) without having to create account on any of those sites.
2. The current data dumps are public and accessible to everyone.

This is how the process is going to be after the change:

1. In order to download a data dump of a certain site, one would be required to create an account there.
2. This is not public and restricts access to the data dumps.

Now some interesting quirks.

1. Suppose one is suspended on a site. Would it mean they won't be able to download the data dump?
2. In 6-8 months/years SE suddenly announces: "Hear hear, we decided to limit the download for users with over 50k reputation, this will surely boost their morale a lot".

So no, it is neither quicker nor more efficient, in the bigger picture. The total opposite.

Answer 5

this is primarily only a change in location for where the data dump is accessed.

No, this is false and misleading. You are instead trying to quietly change the license of the data dump, and you disguise that change as a simple change of data host.

This is not the same as the previously agreed CC BY-SA license, which allows commercial use (e.g., in the event one needs to fork Stack Exchange if the ship sinks, as for example Yahoo! Answers did).

We usually avoid posting on Fridays, but with the data dump scheduled for the end of July, we wanted to share this information with the community as soon as possible.

No, this is also misleading. Here's the real reason: "The changes to the Data Dump were posted on Friday, because you needed to rush out the announcement so that Prashanth could brag about it at a conference [a few days later]!" [AMtwo].

with the heightened ease of receiving a dump

Once again, this is misleading. It's not heightened. It's significantly decreased, since the new way has no command line interface and it's for only 1 out of 184 network sites (which includes their corresponding metas). I've downloaded and uploaded the Stack Exchange dumps dozens of times from/to archive.org, works fine. Downloading is a one-liner:

ia download stackexchange --retries==100

Or to download all images:

ia download stack-exchange-images --retries==100

The new dump method instead requires 920 clicks (=5*184) and still doesn't even include images.

Answer 6

The company cannot be trusted to own the server hosting the data dumps.

You (read here and later: the company, just so I can skip a spammy round of "I didn't do anything") want to make access to the data dumps faster. Delightful! Regardless, you must keep the old, independently hosted archive path, because the company can, and the company WILL pull the plug on the data dumps. When that happens, we must not lose access to at least all past data dumps until that point.

Always worth coming back to Joel and Jeff's time:

Oh, expropriation of community content that... We created Stack Overflow to be against it. If there's anything that's more in the DNA of Stack Overflow than that, I don't know what it is. That's one of our most core things. You can see this all over the place in the design of Stack Overflow.

First of all, from day one, we use the CC-wiki license. And it's basically a license, it says that we don't own the content that's on there, which is why we make those database dumps that are available.

Because we wanted to make sure that if no matter what happens, literally no matter who we sell to, or raise money from, or turn the site over to, and even if they take Stack Overflow, and make it an evil site where you have to pay to look at things and there's pop-up ads and pop-under ads, and you know, dancing chariots of fire that cross the screen and punch the monkey, and, man, I can take so many evil things anyway. And it just becomes a big gigantic spam site.

Doesn't matter because just take the latest CC-wiki download that we provided and go start your own site saying, you know what, this is gonna be the clean version. And I think a lot of people will follow you. We very, very deliberately built Stack Overflow in a way that there wouldn't be any chance of locking and we're pretty much doing the same thing with Stack Exchange.

You are making it very easy to pull access to our own content that brings you profit. Even if we trusted the company now, this would make it not just possible, but trivial, for some future nefarious company leadership to backstab the community. And guess what: we already have the nefarious company leadership in the present.

Disabling data dumps uploaded to archive.org or some other third-party, truly independent service is unacceptable.

Answer 7

Posted Tuesday 2023-06-13 22:32:13Z:

Much has been written lately of the company’s decision to pause the distribution of the anonymized data dump that has historically been posted.

Our intention was never to stop posting the data dump permanently, only to pause it while we begin to collect more information on how it was being used and by whom - especially in light of the rise of large language models (LLMs) and questions around how generative AI models are handling attribution.

You have been engaging on this topic disingenuously for a year.

It was your intention to turn off the dumps a year ago, and now you're trying to make them as inconvenient as possible.

Answer 8

In my other answer, I mostly focus on licensing and why there are materially problematic issues with the current plan. In this answer, my focus is on why the announcement is problematic from a moral standpoint.

The story of how LLMs changed the Company

Github Co-Pilot, the start of the LLM revolution

When GitHub Co-Pilot came out in 2021, a lot of developers saw it as transformative. A portion of software developers no longer needed to Bingoogle their questions, because they had a Co-Pilot that could direct them without leaving the IDE. Stack Overflow, however, maintained their course--focused on Stack Overflow for Teams to monetize the product. The Company did not pivot to begin working on any sort of VS Code extension, or other LLM/AI-powered features.

ChatGPT, the commoditization of LLMs

When ChatGPT landed in November 2022, however, the Company noticed. And the Company realized that they were being left behind. This has lead to a frantic attempt to catch up, but also attempts at kneecapping the competition. Notably, I believe the only AI-powered or LLM-powered products delivered to date are limited to Stack Overflow for Teams, and are not publicly available for individual use.

On March 28, 2023, the Data Dump was disabled at the direction of Prashanth Chandrasekar, the Stack Overflow CEO. At the time, I was on the DBRE team at Stack Overflow, and I was the person who the CEO directed to disable the upload to Archive.org. That conversation ended as thus. This is a direct transcript of our conversation--I've removed a single back/forth for brevity, represented as ...:

PC: If we disable, how long does it take to re-establish the link if we want to open it back up?

Me: Disabling/re-enabling is just a simple switch. It's a scheduled job that runs once per quarter--so er can just turn it back on and/or kick it off ad-hoc anytime we want.

But if we don't upload on schedule, we're likely to have someone notice and ask about it on Meta. So we would need to be prepared to respond--or better proactively explain it on a Meta post of our own.

The Community Team probably just needs to be in the loop to not be caught off guard with the customer service side of it.

...

PC: And remind me, we know who is accessing the data in Archive.org?

Me: We don't really know. The internet archive doesn't require users to be logged in to download it.

PC: OK, then lets auto disable it for now so we don't forget and we can come up with a plan with the community team in the coming weeks.

Just over a month later, on May 10, 58 employees, including me, were laid off. I was no longer able to influence community notification.

Broken Promises

Data Dump Disaster (Data Dumpster Fire?)

A month after that, the Community was surprised when the Data Dump wasn't published per usual, despite the assurances made to me that the CM team be brought in. I take issue with the subsequent statements by Stack Overflow CTO and by Philippe that it was always going to be re-enabled, but I was satisfied with the outcome (the Data Dump was brought back).

In Philippe's announcement, he made two promises:

We will continue to work toward the creation of certain guardrails (for large AI/LLM companies) for both the dumps and the API, but again - we have no intention of restricting/charging community members or other responsible users of the dumps or the API from accessing them.

As part of this project, API users should be on the lookout for a very brief survey that will be coming out (it will be announced here and on stackapps.com) that asks about the features that you most use/would like to see in the API or data dumps moving forward so that we can plan for those, as well as collect general input.

Survey? What survey?

The company did send a survey about the API, without mention of the Data Dump. Speaking for myself--a consumer of the Data Dump, but not of the API--I didn't respond to the survey, and could only hope that another survey would be sent for the Data Dump. It wasn't. As such, the Community was never solicited for feedback on how we use the Data Dump--we've not had our chance to share our user stories that would drive changes to the Data Dump.

I can only assume that the reason Community Users were never surveyed about our Data Dump usage is because the company doesn't care how we use it. Their intent has been clear for the last year--they couldn't eliminate the Data Dump completely as they originally hoped, so they will restrict it as much as they can get away with. The direction to do so comes from the CEO.

Maybe someone just conflated the API and the Data Dump? Seems unlikely that anyone familiar with the two products would conflate the two of them given how dramatically different they are. I've noticed that the API often gets name-dropped when the Data Dump is being discussed, and vice versa. However, they serve very different purposes--one is great for "point lookups" on individual posts, while the Data Dump is best for set-based queries and analysis.

Fast Forward to today...

Senior Leadership has been thinking about this since at least March 2023. In June 2023, Philippe explicitly promised that they were working on "guardrails" for the data dump. More than a year later, that work hasn't been completed--the feature was only sent to engineering teams last week.

Despite the Company "working on" a new data dump process for over a year, it still isn't ready. Additionally, the assurance that the old process would stay in place until the new one is ready has gone out the window.

• By turning off the old process before the new process is ready, the company has broken a promise made just last year.
• By moving the data dump off of the Internet Archive, and onto Stack Overflow hosted infrastructure, the company has broken a promise made by the Company's Founders to archive the data with a 3rd party to ensure access to the data stays open.
• By surveying only API use, and not Data Dump use, the company has broken another promise made just last year to "[ask] about the features that you most use/would like to see in the API or data dumps moving forward"--while this was done for the API, it wasn't done for the Data Dump.
• By attempting to place additional restrictions (ex, non-commercial use), the company is violating their own Terms of Service which require the Data Dump to be licensed under the CC BY-SA license.
• By failing to use Community feedback in creating this data dump plan over the course of the last year, the Company has alienated the content creators who contribute the data that makes Stack Overflow unique.

My proposal for moving forward.

Keeping in mind the company's core values...

The Company has fallen woefully short on the values of "Be transparent" and "Keep community at our center." I would propose that the company take the following steps, which would be more in line with the Company's core values:

• Recognize that the Data Dump is a product intended to support open data, and the rights of the content creators who maintain ownership & copyright, while also licensing creations liberally and openly.
• Embrace a problem-oriented solution, where features and decisions can be mapped to a user story.
• Re-enable the existing Data Dump upload to the Internet Archive until such time that a suitable replacement is complete and ready. Ideally, the two methods would overlap for one quarter to allow a transition period.
• Survey the users of the product (Community users, researchers, etc who use the Data Dump) for how we use the current Data Dump, and what current or future "features" are important to users.
• Share with the community any requirements that the Company has, in addition to the end user requirements.
• Improve transparency around licensing, and how licensing has impacted decision making. (Responses to this question show many contributors are deeply committed to open data, and permissive licensing of their content.)

Harsh reality

Every Question & Answer on Stack Overflow is licensed under the CC BY-SA license, which allows commercial use, and already requires both attribution and "share-alike" to maintain the openness of the data. The company seems focused on restricting commercial use, which is not only futile and ultimately an impossible game of whack-a-mole--but also a violation of license terms. The company's only rational, legal basis to prevent abuse of the data is to pursue use that violates the BY-SA license.

A Data Dump will end up on the Internet Archive, even if the Company doesn't post it.

The Company will get sued if they attempt to prevent commercial use of data that is explicitly licensed for commercial use. And the Company will lose such a lawsuit.

The Company has further hurt their relationship with content creators by continuing to repeat the same mistakes year-after-year.

Contributions are already on a precipitous decline, both overall, and from high-rep users; one can assume this change will hasten decline for contributors who are passionate about their creations' licensing & future use. I sometimes think of "votes cast" as a measure of engagement (people who care enough to give feedback), or helpfulness (This worked/didn't work), or even just presence--the number of votes cast is also in free-fall.

Answer 9

A few things to point out:

1. The text in the screenshot around requesting access is not consistent with saying that the CC BY-SA license is unchanged. The Creative Commons has a FAQ entry about entering into separate or supplemental agreements with users. Specifically, see the section on supplemental agreements, which states that "problems arise when licensors design those terms or arrangements to serve not as separate, alternative licensing arrangements but as supplemental terms having the effect of changing the standard terms within the CC license." It goes on to say that, to avoid confusion, Creative Commons "must insist that in these instances licensors not use our trademarks, names, and logos in connection with their custom licensing arrangement." One solution would be to license the data dump CC BY-NC-SA. However, this would require an update to the Terms of Service. In addition, there are still open questions about whether creating the data dump is sufficiently creative to allow it to be licensed separately from the works contained in the data dump, which all allow commercial use.
2. There is no explanation as to how you require your "partners in socially responsible AI" to comply with the CC BY-SA license attribution requirements. Enforcement is the responsibility of the work's author, and no clauses in the terms of service or any other agreement authorize the company to be our agent and enforce our rights. In addition, Creative Commons has an entire section of their FAQ dedicated to AI, including a nice flowchart. The CC license, including the attribution clauses, are not triggered if an exception or limitation applies. Specifically, with respect to training generative AI, Creative Commons believes that training is fair use. Other groups also hold similar stances. Until there is judicial precedent or legislation that says otherwise, it's unclear why the company is taking a different stance.
3. When do you plan to provide more information about your current stance as well as the planned future stance on data scrapers is? Personally, I'm interested in this topic. For example, I've inquired about robots.txt and other technological measures to block crawlers and how changes are decided and communicated. It would also be good to know if this stance change will allow the company to take action against all abuses, such as the YouTube channels that are walking a fine line (and perhaps crossing the line) with compliance to the license and monetizing content. The current stance is that the company requires authors and content owners to protect their work, so it would be interesting to understand where the line is now and the basis for enforcing our rights.
4. Although I don't have a problem with SE hosting the data dumps in their own infrastructure, some improvements would make this better:
   • The current implementation requires that a person have an account on each site that they obtain a data dump from. This can "leak" PII to network moderators for the entire network. Adding the ability to download dumps from the network profile should be added.
   • In addition to moving single-site downloads to the network profile, allowing bulk downloads from 2 or more sites in the network or the entire network would be good.
   • The API should be updated to allow for programmatically obtaining download URL(s) for data dumps.
   • One of the reasons for posting the data dumps on a third-party location was to ensure their availability should something happen to the company or network. Consider making older data dumps available on third-party locations, such as the Internet Archive. After some length of time (perhaps 2 or 4 quarters), submitting the data dump to one or more third-parties and allowing users to do the same with no risk.

Answer 10

Having assumed you were acting in good faith, I spent a full working day going through the third draft, pointing out wording that was likely to result in… well, *gestures at this webpage*. And I don't think I'm the mod who spent the most time on this.

You said you would read my notes carefully. Obviously, you didn't. In the section labelled “for PR purposes”, you stopped at the first bullet point! The second:

• explicitly mention the measures you have in place "to ensure [our] contributions outlive [the company]".

anticipates Andras Deak -- Слава Україні's answer. There's a reason I wrote those things!!!

I'd make this post a systematic blow-by-blow of avoidable complaints, but going through the feedback from other moderators can't be done in a public space without their permission. Suffice to say, most of what's written in the other answers here is stuff that moderators raised weeks ago, that we were told would be addressed.

---

I also wrote:

Can we give feedback on the alt text for those images, even though the images aren't ready yet? I don't like editing official Stack Exchange announcements, but also, images need proper alt text.

Firstly: here's your alt text:

The proposed location of the data dump access within the account settings; please note that some of the placement/layout/language is subject to change.

This is not acceptable for what the image is – but secondly, this image is the only place in this post that says:

• that people who download the dump are required to explicitly agree to something; and
• the text of that agreement.

Here's the text:

Data dump access

You can access this site's data for non-commercial use. For commercial purposes, please contact us.

---

A new data dump is available every three months. Learn more about the data dump process.

I agree that I will use this file for non-commercial use. I will not use it for any other purpose, and I will not transfer it to others without permission from Stack Overflow. I certify that I am not downloading this file on behalf of my employer, for use in a for-profit enterprise. I have read and agree to the Terms of Service and the Acceptable Use Policy, and have read and understand Stack Overflow's privacy notice.

This is the exact problem that was raised on 2024-07-09. What was going through the pseudo-mind of your dysfunctional organisation when your colleagues in an actual decision-making role decided to rewrite the entire announcement to remove all mention of the federal crimes the company plans to commit, without changing the process, but include that information in a screenshot, which you hid from the moderators who you asked to review the process.

What was even the point of the review, then?

This wasn't even subterfuge. It was gross incompetence. To the extent that your colleagues' jobs are to steward the Stack Exchange network – a community resource ­– they are not good at their jobs.

To the extent their jobs are to protect the company's number-goes-up, they are also not good at their jobs. Number has gone down.

---

I was going to demand you schedule a meeting to read a thing to the people who made this decision, and then I noticed that you already did. Have you read its predecessor to them? You should.

Philippe, you care. Somebody clearly doesn't. Please stop letting these buffoons hide behind you. Let the cowards take responsibility for their own terrible decisions.

I keep believing you because you care. At least – as a favour to me –, make it clear when you're not in a position to call the shots, so I can avoid wasting my time. The alternative is that I stop believing you.

Answer 11

I have to join this discussion a little later due to being mostly offline in the last few days.

Let me start with a personal consideration. I won't sugarcoat it. You have reached a new low. This is pathetically bad.

In the last months I have asked you multiple times if you were trying to remove anonymous access (this is just one example) and you conveniently ignored that (like you ignored my questions about the options in the data request page...). Now I come here to see I was right.

The company is trying to put a login wall on the availability of the data dump. Given this post's passive aggressive threat that "when you breach the agreement that you make when downloading the dumps file, we do have the option to decline to provide you with future versions of the data dumps," I suspect the company thinks they can watermark the dumps to be able to identify who posted a downloaded file to a public facing site... and not get their attempt utterly destroyed the second that the community - mostly made of devs - decides to revolt. Also, please remember that the worst option is RPA, and despite you thinking you can detect that easily... trust me you don't want to force people to organize.

I will forego here any argument about the legality and morality of the company actions. I think others have already done a great job.

I will instead point out what - in my opinion - the company are actually trying to do because I didn't see this mentioned a lot.

The company's recent actions clearly point at two parallel intents:

1. The company want to artificially inflate the userbase numbers. Constantly showing a "pwetty plez, make account" prompt in the face of anonymous users clearly shows how you hope to increase user "activation." There is no doubt about that.

   Yet what I am not sure about is if everyone in the company really thinks this is going to work or if this is just a "mad cell" trying to scam the top management with bigger numbers that won't translate to actual contributions: (Look at these shiny numbers! 1k more users on the site... Yeah, they didn't post anything, but that is 1k more users still!!! Our plan to annoy them to the point of creating an account and never posting anything worked great!)

2. It is also clear that the company doesn't care at all about the site purpose. The only value Prosus sees in the site is selling the data / using the data as training material. Considering the recent OpenAI partnership, I think it is pretty safe to assume that Prosus is desperate to try to get the exclusivity to the training material the community is making available and is willing to do whatever it takes to that goal, at least until some clamorous mishap ends this story in disaster (and to be honest I think they are really getting there).

   Like before, I am also not sure of the actual reality here: does Prosus really think they can restrict access to the content - freely reproducible under a CC BY-SA license - without someone making it available somewhere else? Or is this just pretending to be able to in order to mislead their OpenAI partner into thinking they have the "strategic advantage of exclusive access to Stack Data"? To be honest, I don't care; for us users the result is still the same.

I think there is only one reasonable course of action. Starting today, I will officially stop contributing new Q/A content to the network and performing any curation activity. I will only stay active on posting on Meta.SE.

Answer 12

We would really rather users do not upload the file to archive.org or similar data pile sites. Assuring the viability of the Network takes resources, and companies that profit off the back of the work of this community should feel an obligation (“socially responsible AI”) to give back to the communities whose work they use to create the models that they are marketing commercially. Our hope is that because the process for individuals to request the dumps is lightweight and quick, you won’t feel the need to undermine these efforts to encourage commercial re-users to contribute back.

You (the company) don't understand.

The Creative Commons Share-Alike license is not anti-commercial.

All generative AI that absorb CC BY-SA works aren't complying with the license. This is my opinion; I'm aware that the CC org itself seems to disagree and some courts might rule differently. But mine is probably the common opinion of most people releasing things under that license. (Surveys to the contrary would be interesting to see!)

But we contributing members of the SE network are not against the commercial exploitation of our content. The SE company profits off our contributions! So can other companies, so long as they comply with the license.

There is no implication of "giving back" in the CC BY-SA license, instead it has a legal requirement to license derivative works under the same/similar license. As long as you do that, if you can make a profit from our contributors, then the CC BY-SA license is an open invitation to do so! We contribute to our SE sites because we want to help people learn about hundreds of topics. If companies can do so in new innovative ways, while complying with our license, then that's great. There is no implication or expectation that they somehow contribute back to the SE community in the process.

Answer 13

I have many, many thoughts on this, so this answer might end up being a bit fragmented. I know this isn't novel among the other voices here, but I think this is a very, very bad idea.

Self-Interest & Hosting Burden

Something that dumping to the Internet Archive does is remove Stack's self-interest from the equation of hosting the data. I.e. Stack no longer pays for the burden of exfiltrating that much data to every user who wants it, and Stack isn't in unilateral control who can or cannot access my content, in accordance with its license.

I don't mean to suggest anything about Stack's ability to pay for bandwidth costs– my point is that by taking sole ownership of hosting the dump, especially when there's clearly expressed interest in this very question for why they don't want the dump to be accessible, the Company is taking on paying for something that they don't really want to support in the first place. In other words, this move creates a clear & tangible incentive for the Company to discontinue the dumps whenever the cost begins to seem unreasonable, even more than already exists.

That's bad.

The reason the dumps were done to begin with was a goodwill stake in the ground, a tangible means of saying "the Company is a steward of the data, not its owner or arbiter". By putting yourselves in control of who gets to exfiltrate the data, even if today it's wholly unrestricted, you've created a massive temptation to decide who does or doesn't get to have it, or to disable it altogether.

In my view, completely regardless of any tangible benefits like download speed, it's a terrible, awful, no good, very bad idea to put yourselves in that position if you have any inkling of care for the goodwill the dumps were designed to facilitate all those years ago.

OverflowAPI & The Dump

This is a little bitter, but it's staggering to me that the Company apparently thinks that its OverflowAPI product offers so little value that the raw data dump would even register as a potential threat.

Does the Company really believe this little in its own engineering capacity?

The data dump is giant, unwieldy, unfiltered, and delayed – if OverflowAPI is so bad or limited that a client would even consider the dumps as a reasonable alternative, then there's incredibly vast room for improvement.

The dumps should be 99.99% orthogonal to the existence and value of OverflowAPI. Your product must offer more value than the data alone, precisely because the data is already out there– that's the inconvenient (for the Company) truth of the CC-BY-SA license that's baked into the foundation of the site.

Hiding the dump isn't going to change that.

Hypocrisy of Expecting "The High Road"

The Company is stomping on a lot of goodwill to protect its interests here (it's seemingly been a theme lately).

If you, the Company, are willing to trash the years of goodwill represented by the openness of the Data Dump; willing to take arguably one of the most significant remaining pieces of consistent goodwill offered to the Community, and crush it, lock it away, in order to protect your own interests...

...then how can you possibly expect other companies to take the proverbial high road and pay for your API instead of scraping the site or using the content dump that they have a legal right to use under license, if its not in their interest?

It feels absurd to come all this way, stomp all over what the Data Dump stands for, clearly and demonstrably misunderstand what the license even means and allows, and then still, for some reason, expect other companies, who care about their own bottom line(s), to "take the high road" when you arguably won't even take it yourself.

There's a reason folks are really bitter about this move.

Answer 14

We, the community, want to contribute content under a Creative Commons Attribution ShareAlike license, to share with the whole world. We're inspired by Wikipedia and other free culture initiatives. Stack Exchange, Inc is making it difficult to access the knowledge that we want to be freely shared with the world, and that's a threat to our mission. We trusted you, Stack Exchange, to steward this data well, even though you never paid any one of your users or moderators for our contributions, not even one dollar. Freely you have received, freely give: that's the mission we all signed up for.

Of course, we want you to make money! I also want all the contributors and moderators to make money. We don't hold you responsible to work out how to pay us for our work, so don't restrict our knowledge base and our data dumps because you are not making as much money as you would like, or because you envy NVidia, whose market cap is $3.18 trillion, more than the entire market of many countries.

I'll quote Zoe:

There's no way around it anymore now that you [SE] have pushed this through in spite of internal warnings; you [SE] have made Stack Exchange, Inc. the biggest threat to the sustainability and future of the community. genAI is in second place, because the ongoing damage you [SE] are doing to the community, and to data archival and preservation efforts beats any damage genAI could do by scraping the data. While this hasn't been said anywhere, it's clear to all of us that the only reason you [SE] are doing this isn't to protect the community, but to protect your [SE's] revenue

Answer 15

The data dump is a hedge against the company failing or turning evil. It was that promise to keep the content available to the global public that made a lot of us willing to donate our time and expertise not only creating that content, but curating it. Placing the data dump behind a link where the company can arbitrarily deny people access breaks that promise. Even if the company were currently trustworthy, it would still be a problem because we can’t know whether the company would remain trustworthy.

How long before there is a reputation requirement to access the data dump? I assume access will be blocked from tor to prevent people from getting around being denied access. Yeah, this is going to work just fine in the short term. Stack Overflow has plenty of meat left on its bones, so you will be able to dine off its corpse for probably many years to come. My prediction is that the folks who are willing to donate to the company in exchange for internet points are not going to be as reliable partners as the people who do it to build a repository of knowledge for everyone regardless of whether they can afford to subscribe to Google’s AI service.

Ending public access to the data dumps will be a hard thing to undo. I hope you looked at the data and can see that the community still hasn’t recovered from the strike. This could very well kill the golden-egg-laying goose. There are ways to monetize the data dump without breaking the promise to the people that created the value you’re trying to sell.

Answer 16

I understand that this file is being provided to me for my own use and for projects that do not include training a large language model (LLM), and that should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump.

This has moved from questionably (il)legal to almost certainly unethical.

Most of the issues surrounding Section 2(a)(5)(C) of the CC BY-SA license, which states that "you may not offer or impose any additional or different terms or conditions on ... the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material" have been cleared up.

The key changes to the wording make it clear that the downloader can exercise all of the rights granted to us by CC BY-SA. The only condition is that if we use the data in certain ways (such as training an LLM), we may lose access to further updates to the data set. This seems to be permissible, since it's already established that a distributor can establish criteria for distribution and is not obligated to distributed CC-licensed content that they hold.

The only open licensing/contracts/legal question that I see is the last part: "should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump".

Although the Creative Commons licenses allows the licensor to make preferences and non-binding requests, this seems to be a clickwrap contractual obligation. On the surface, it doesn't impose any additional terms or restrictions on sharing the version of the data dump I have, it does require me to establish control of or otherwise accept the responsibility of third-parties in order to guarantee continued access to the data dumps. This seems like a questionable contractual term, at best.

Depending on how current court cases go, this may also become a moot point. My understanding is that copyright law generally takes precedence over contract law in many, if not most, cases. If the US courts find that training an AI model is fair use in general or a particular instance of training an AI model is fair use, then this agreement could become meaningless.

The claim that there is a "heightened ease of receiving a dump" is also blatantly false. At least in the current state, not only is it more difficult to receive the data, there are also privacy implications in that a downloader needs to have an account on each site and expose their PII to every mod on the network. The fact that main and meta dumps are separate also increase the number of downloads. There are solutions to this, such as moving the dumps to the a person's Stack Exchange network profile instead of (or in addition to) individual sites. Claiming that this is easier until it actually is easier is wrong, though, especially since we don't know how many data dump releases it will be until it truly becomes easier to access the data.

More broadly, though, these changes violate some of the fundamental principles of open source and the commons.

Consider some words from the early days of Stack Overflow:

Everything contributed to the Stack Exchange network of websites is licensed under Creative Commons Attribution - Share Alike. This means it belongs to everyone, and can be freely reused (even commercially!), so long as it is follows our simple rules of attribution. That's our contract with the community -- it's your generously contributed content that makes these websites worth visiting in the first place!

To emphasize: freely reused, even commercially, by everyone.

This is the promise that was made that led us all to contribute.

Some people may not like how people use their work. However, a principle of open source is a lack of discrimination. We do not discriminate based on the person, group, or endeavor. The work is out, in the open, for everyone to use.

I do understand the company wanting to monetize the content to the extent possible. However, this isn't the way to do it. Take a lesson from Wikimedia.

Instead of adding terms and conditions to the data dumps, sell more frequent data dumps (if that's something that the AI companies want). Instead of quarterly public data dumps, allow people and companies to buy monthly data dumps.

I highly suspect that the company is doing something similar with OverflowAPI. Without seeing documentation, I suspect that the API isn't that different from the public Stack Exchange API, except maybe with even higher rate limits and perhaps a streaming API.

The underlying problem is that the company is unilaterally making decisions on how to manage, share, and protect user-contributed data without involving the people who own the data. There is no significant effort to understand our concerns and hear out different viewpoints about our concerns on who is accessing our contributions, how they access them, and what they are doing with them.

The company doesn't put any significant effort into the data dump, with respect to selecting, arranging, or otherwise curating the content. It's a raw dump of the data that people have contributed, and it's not even provided to consumers in the most usable way. Some people have even found errors in the data.

I highly suspect that, when you wanted to start to put these "guardrails" up, if you came to the community and said things like:

• We want to reduce the free API daily request quote from 10,000 requests per day to 2,500-5,000 requests per day and reduce the number of requests per second from 30 to 10, but we will work with trusted community members to ensure that public applications will continue to work.
• We will slow down the production of free public data dumps to once or twice a year in an attempt to encourage commercial adoption. However, if you are a community member or academic researcher, we will work with you to review your use case and deliver more frequent data.

We wouldn't be where we are today.

Answer 17

So there's a few 'transformative' versions of the datadump that respect the CC licence, and exist specifically due to the open nature of the network - Brent Ozar's MSSQL conversion comes to mind. There's an older dump on bigquery by google (who is a SE partner) which appears to have been last updated in '22 . There was also an offline SE instance by Kiwix , which is (was?) officially supported by SO inc and has it listed as a partner.

All of these examples are in the spirit, and letter of the rules, and two of them are officially supported by SE. They are either educational (Brent), non profit (Kiwix) or in the case of google a current partner

Considering they respect the CC licence but downstream users may not, how would this new policy affect someone who wants to share the data for non profit or educational use but could potentially have their work used by LLMs with or without their knowledge?

Answer 18

Is the download going to be from a torrent? When I have downloaded the dumps in the past, the only way I've gotten it to work was through bittorrent. The dumps are just too big to download through other protocols in my experience.

Answer 19

The data dump should be easily accessible, that's what open source data is all about. The fact that you're putting roadblocks in the way of researches will just make legitimate users not access the data. I'm completely against the "ask for access" mentality because there's no guarantee you'll give access in a timely manner, if at all.

Oh, and as Franck's post points out: you are in fact relicensing! But I never gave you permission to relicense my contributions, and the license was irrevocable when I granted it.

Answer 20

Today, we are announcing some changes to the data dump process. I’m going to start with an important statement: this is primarily only a change in location for where the data dump is accessed. Moving forward, we’ll be providing the data dump from a section of the site user profile on a Stack Exchange profile.

There are a number of reasons for this: first, this is an attempt to put commercial pressure on LLM manufacturers to join us and our existing partners in the “socially responsible AI” usage that we’re advocating for - to get them to give back to the communities whose data they consume.

Second, we want to help make the process of accessing data dumps quicker and more efficient. While the Internet Archive has been a great partner to us, as you may know, both internally and externally, people have encountered challenges with uploading and downloading the dumps with any reasonable speed.

Lastly with the heightened ease of receiving a dump, we’re curious to see what community members will build with this information.

It finally happened, albeit it took 1 year and 1 month from when you first tried it but nevertheless, here we are.

Always talking about the "greater good" or "protecting users", just so unbelievably disingenuous and extremely insulting, as though your buzzwords will convince or fool us.

You don't care about what people have or will build from this, this is nothing but an attempt to push companies towards your commercial offering - whatever shape that may take.

When organizations are able to skip out on their obligations to contribute back, the whole internet suffers. Without Stack Overflow as a resource, many of the world’s millions of technologists would not be able to find the answers to their programming questions. And we know that these organizations are not contributing voluntarily - so putting this light commercial pressure on them is the best method to encourage socially conscious behavior. It’s important to say that when you breach the agreement that you make when downloading the dumps file, we do have the option to decline to provide you with future versions of the data dumps. But we really don’t want to have to do that.

Utter hypocrites.

Management should take their own advice first and foremost considering how they treat their own power users, you know, the ones that actually contributed towards making this resource what it is.

First, our product team has regretfully informed us that it will not be possible, within the existing time constraints, to bundle the dumps for a site and its meta together. So if you want to get the download for a site and its meta, you will need to do the request on both sites. We know the initial announcement implied that the two files would be bundled together, and we’re sorry that won’t be the case right now. That bundling was initially assumed to be a low lift, but it turned out not to be, so was taken out of scope for this release.

My answer from when the June 2023 data dump was flagged as missing seems particularly prescient:

We're quickly scrambling because this would be an amazing opportunity for us to make some more money off your backs.

We're probably going to degrade the experience for people within the community who used to use these dumps because we need to somehow get companies to pay us to use this data in a more friendly way.

Answer 21

The "pretty please don't use the data dump commercially thank u" checkbox and all the potential legal issues around it have been mentioned ad nauseum already, but given the WeAreDevelopers summary that was posted last week¹, and how this change's primary purpose - according to the post - is "[...] to still be accessible by the majority of users while restricting access to those using them for commercial purposes", it's worth drawing attention to this part of the announcement:

I’m going to start with an important statement: this is primarily only a change in location for where the data dump is accessed. (emphasis not mine)

To quote Tyler the Creator, "So that was a f***ing lie." If you as a company have ulterior motives regarding the changes you make, at least try not to brag about them at a conference?

I won't repeat myself and preach to the choir more than I need to, because quite frankly everyone else has already made the point far more eloquently than I can. But to me, this confirms in my mind what everyone is afraid of with these changes. Stack Exchange is seeking to add downstream conditions beyond the CC BY-SA license that the data dump is licensed under. This is potentially a breach of the license, and can potentially result in the license being terminated.

^{1 And very conveniently, was not featured! Even though last year's post of the same event was featured for over 3 weeks.}

Answer 22

Frankly I'm impressed by all those assuming good faith.

This announcement is the worst form of gaslighting going. Don't try and pretend any of this is being done on our behalf.

Answer 23

A bit late to the party, but...

This could create legal problems for Stack Exchange.

Specifically, this has to do with the CC BY-SA license's requirement to remove one's name from content if one asks (i.e., post dissociation).

As quoted in the Creative Commons FAQ that is linked to in the post:

if the licensor does not like how the material has been modified or used, CC licenses require that the licensee remove the attribution information upon request [...] in 4.0, this also applies to the unmodified work.

When a post is dissociated from a user upon them making a request under said clause of the CC license, prior data dumps are not updated to reflect the change as they're historical content.

If the data dumps are hosted by someone else, this is perfectly legal, as Stack Exchange wouldn't have control over them and the host would technically be a different "licensee" under the above quote. However, if the data dumps are hosted in-house, then Stack Exchange would be under legal obligation to go through prior data dumps and remove the requester's name from them since they're the same licensee.

Having someone else (e.g., the Internet Archive) would remove the need for SE to do this on every data dump and thus greatly reduce the workload on SE staff. Yes, I'm aware that "there's no such thing as true dissociation on the Internet", but this is a legal issue and thus something SE's obligated to do even if it doesn't technically matter.

Answer 24

If data dumps are not available, we can't check that your AI is in fact "socially responsible" as you call it. I don't trust you that much.

You claim that attribution is non-negotiable: https://stackoverflow.blog/2024/02/29/defining-socially-responsible-ai-how-we-select-api-partners/

but AIs are known for misattributing/not attributing things. So if you can't prove that your AI does not have this problem - by making data dumps available, so we can check all of AI answers against all the data it was trained on, I call your "socially responsible" label a fraud that you are trying to hide by disabling data dumps.

Answer 25

Please clarify "and" in the new checkbox wording

this file is being provided to me for my own use and for projects that do not include training a large language model

Is it just me, or "and" is a bit ambiguous here?

I assume you mean for projects that do not include training a LLM, unless it is for personal use only, but this can also be interpreted as only for personal use, and additionally this use must not include training a LLM.

I'm fairly sure you mean the former, but either way I suggest updating the wording to one of those two options.

---

Assuming you meant the former:

The new checkbox wording is much better

The reason why I care about the data dumps is because they (in principle) allow bootstrapping a new community/site elsewhere, if SE becomes evil (or too evil to use). I don't care about LLMs being trained on it.

I even thought about proposing this exact thing, but figured SE wouldn't listen.

Answer 26

this is an attempt to put commercial pressure on LLM manufacturers to join us and our existing partners in the “socially responsible AI” usage that we’re advocating for - to get them to give back to the communities whose data they consume

You do understand that creating an account and contributing are two different things, right?

Answer 27

Well, looks sane! As an AI developer myself I'm totally for this type of licensing! And - to avoid the bots - my offer is to adopt the Wikipedia's approach: they do offer regular dumps pre-made by them for download, so there's no any need for scraping anything! Cloaking the data from AI or poisoning it sounds like a 21th century luddism for me - why protein neural networks can learn from something but other can not? It's OK!