The company's commitment to the data dumps, the API, and SEDE

Question

Many words have been written around the company's commitment to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE). Much of that text can be confusing or conflicting.

So in order to make clear the company's position, we're publishing this note.

The company is committed to the long-term (foreseeable future) survival of the data dumps, the API, and SEDE. We will continue to maintain them, and assure that community members have free access to them for legitimate usages that support the community, broadly construed (including for study in classrooms, for instance). We retain the right to place guardrails around them to ensure that companies constructing language models, etc, are charged for access, but community users, including the Charcoal anti-spam network and similar projects will continue to have free access.

Answer 1

(Note: the quoted sentence was edited after I posted my answer. I am leaving this answer unedited and undeleted, so that it is available for folks reading the revision history.)

---

From the original post:

Many words have been written around the company's commitment (or perceived lack thereof) to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE).

I find the inclusion of the "or perceived lack thereof" parenthetical to be incorrect, and unnecessary. At best, it is an unnecessary "dig" at members of the community who questioned the missing data dump earlier this year.

As I mentioned in my answer on that question, I was the individual who pressed the metaphorical & literal button to disable the data dump upload job. As such, I take that dig as directed (at least partially) at me specifically.

Because I find that statement as both factually incorrect and targeting me specifically, I do feel the need to address that comment.

Leading up to the disabling of the Stack Overflow Data Dump, I had a number of conversations with Prashanth, specifically around the Data Dump, the API, and SEDE. I have specific firsthand knowledge of the decision to disable the Data Dump, as well as some of the research that went into that decision.

I believe that the implication made by Philippe, that the Data Dump was never going to be disabled long term, is both untrue, and damaging to my reputation as a data professional, as a dutiful employee of Stack Overflow, and as a member of this online community.

Among the questions I fielded from Prashanth included:

• Can we pull down the existing data dump from the Archive?
• Can we limit who can access the Archive?
• How do we limit what data people can download from SEDE?

None of those are direct quotations, mind you. They are simply based on my memory of the calls we had discussing the topic.

March 28, 2023

On the morning of March 28, Prashanth contacted me via Slack DM to ask me to disable the data dump, and mark it as not to be re-enabled without his specific approval.

During that conversation Prashanth said (this is a direct quote, with my emphasis adding italics): "how long does it take to re-establish the link [to Archive.org] if we want to open it back up?"

As part of my response, I said (also a direct quote): "But if we don't upload on schedule, we're likely to have someone notice and ask about it on Meta. So we need to be prepared to respond--or better proactively explain it on a Meta post of our own. The Community Team probably just needs to be in the loop to not be caught off guard with the customer service side of it."

I was assured that the good folks on the Community Team would be informed and given the chance to communicate, after which point I disabled the data dump upload job.

...IF WE WANT TO OPEN IT BACK UP

I don't think it's necessary for me to say any more. Folks who questioned the company's commitment to the Data Dump were not only justified to ask questions, but they were correct.

The company was NOT fully committed to continuing the Data Dumps, nor were they committed to involving the community in that decision, nor was the promise to me fulfilled to work with the CM team to ensure things were properly communicated in advance.

I am thankful that the company has reversed the plan to disable the Data Dump, and that they are now committed to ensuring they continue going forward. But it is not true to imply that commitment has been unwavering, or that people were incorrect to doubt them.

Answer 2

I echo starball's statement, but I want to elaborate a bit more than a comment would allow.

The statements of "long-term" and "foreseeable future" are incredibly vague and often inconsistent. For me, long-term is measured in years and foreseeable future is weeks and maybe months. That's not an insignificant difference.

When added to the comment in Discord where these mean until "the people involved are no longer with the company", that could be days. Consider a recent history of layoffs at the company. We don't know who "the people" are and there are no reasons to believe they won't get a better job or be laid off at any point in the future.

We deserve actual long-term commitments. And if - or, rather, when - situations arise where something about the availability of the data (via the dump, API, and SEDE) changes, it needs to be done as a discussion around the problems, potential solutions, and rationale rather than silently making changes until it is randomly discovered.

Answer 3

Many words have been written around the company's commitment (or perceived lack thereof) to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE). Much of that text can be confusing or conflicting.

Here's your chance to correct them. As in, clear the air around what was said.

Offer a comprehensive, clear, detailed catalog of events that led to the removal of the SEDE data dumps and what the decisions were around it. Otherwise, there'll always be something said about it that can be seen as confusing or conflicting.

If you want more trust, you start by being honest. With just this blurb, I don't know how honest you're actually being. I mean, we can wait...watching the flag count grow isn't really impacting my day-to-day...

We will continue to maintain them, and assure that community members have free access to them for legitimate usages that support the community, broadly construed (including for study in classrooms, for instance).

The only illegitimate usages that you could legally contest would be if someone were to violate CC-by-SA. As in, so long as someone is sure to attribute where they got this content from, they could do whatever they wanted with it. That's kind of the whole...spirit...of Creative Commons. I may not be a lawyer but I don't have to be to know that.

One more time, for the folks a bit higher up the org chart: if I have the data, and I abide by CC-by-SA (versions 3 or 4 or both), I can do whatever I want with it. Is that clear?

We retain the right to place guardrails around them to ensure that companies constructing language models, etc, are charged for access, but community users, including the Charcoal anti-spam network and similar projects will continue to have free access.

Sure, but those guardrails are going to be flimsy. Again, as long as someone abides by CC-by-SA, you don't really have a dog in the fight. To complicate things, if some actor were found to be doing this, this would imply that you have resources to fight these as DMCA requests, which in the past, you have explicitly stated that the content owner could only do that. How do you plan to rectify this confusing legal position? We're in serious and shaky territory with your need to monetize this data somehow and your legal capacity to be able to enforce it. (Although if you pulled a Red Hat that would be quite hilarious in all of this.)

Answer 4

If you are really committed to maintaining the data dumps, free API and SEDE, you don't need to use phrase "foreseeable future". Probably not even the phrase "long-term".

It just sounds like a good excuse for some point in time where you will undo that commitment and then you can just say "We never said this will not change".

Stack Exchange network was founded on community work and under the assumption that this work will be forever available to the community, no matter what happens to the company. Reference The Data Dumps in: Moderation Strike update: Data dumps, choosing representatives, GPT data, and where we’re holding

If the company ever tries to pull the data dumps again, the community response will be just the same as it was now. If the company is not willing to unconditionally commit to our ability to have access to our content, how can you expect that people here continue their participation by providing new content.

---

P.S. If someone at the company ever start wondering again, why is participation (particularly answering) in decline, you can freely point them to data dumps incident and vague wording in this post that is not inspiring any confidence.

And yes, trying to sell our content to the AI companies which violate CC license and attribution is not exactly helping participation either. Especially, when such move was not being discussed with the community first. Is SE [going to be] selling our content for AI model training? And what exactly does "reinvest back into our communities" mean?

Answer 5

I will avoid further commenting on the issue described by AMtwo and the implications in the first version of the post about the "perceived" lack of understanding from the community. Just know that umpteenth subtle bashing hasn't gone unnoticed.

I will instead focus on the core of this post.

The company is committed to the long-term (foreseeable future) survival of the data dumps, the API, and SEDE. We will continue to maintain them, and assure that community members have free access to them for legitimate usages that support the community, broadly construed (including for study in classrooms, for instance). We retain the right to place guardrails around them to ensure that companies [cut] are charged for access, but community users, [cut] will continue to have free access.

I can't help but notice how you keep mentioning the requirement that "(only) community members have free access". The more I think about it, the more it seems evident that you are trying to find a way to require some form of login in order to access the data and the dump. All while being purposely vague on the timing so that if you can't devise some legal trap to stop companies from just becoming users you can still go back and block the access in about 6-8 time-units from now.

Basically, I can't help but read the message as:

We will continue the data dumps, the API, and SEDE for 6-8 something. We are looking for a way to require login to access the data so everyone else is locked out, but we couldn't (yet) figure a legal trap that wouldn't allow a company to just make an account or some user to just repost the dump elsewhere because of that meddling Creative Common license.

Because of this, I can't see this post as anything else than a vague half promise, and can't really upvote it as it stands.

I will also point out that as Jon Ericson noted on his blog, you probably are already late to the party - Stack Exchange content has already been scraped. Not only ChatGPT probably already includes Stack Exchange datasets up to 2022, but multiple datasets that don't seem to derive from the dump are available online. Just go on Hugging Face or other similar sites and have a look.

All you can hope to achieve by restricting the API/data dump access is still getting scraped by good old web crawling.

---

PS: Once upon a time Joel & Jeff used to make fun of that other site, the one that required you to log-in to see the answers... Maybe you could buy them and become one big "Stack Exchange Experts", because I don't see another way to prevent accessing data that is freely available on the web.

Answer 6

I asked in the Meta Discord about what "foreseeable future" means, to which Mithical replied:

As I understand it, until the people involved are no longer with the company.

, which I don't find particularly satisfying. It's like "we've learned our lesson, but we won't make sure that that lesson is passed down".

What exactly does "foreseeable future" mean here?

Answer 7

I'm rather concerned by the 'two' narratives - and the disconnect between what was publicly released, and honestly the seemingly more credible version we're gotten from current and former staff.

We were told that during the acquisition that the advantages of it were SE would be in a financially more stable position and that it would retain its independence to an extent.

If it's true that the decision was initiated by Prosus - we have a situation where core values of Stack Exchange can be compromised, on the hope of potentially making a profit off said data. This has rarely actually resulted in profit, it seems, looking at the various rounds of downsizing, and the inability to invest in many long running community feature requests.

I hate to break it to y'all, but Google's scraping your data and they're big enough they can kill a site that doesn't comply. OpenAI works with Microsoft and Bing and search engines scrape data. Most of the larger generative AI companies have no incentive to buy what they can just take. While there's lawsuits over theft of art and literature, it could go over either way - and SE might not be able to outspend bigger companies even if legally in the right.

It feels like the company might have chosen to sell its values, and spend already overdrawn community capital in the hope of a payoff. There's no profit here, and there may be no guardrails in place that could lead to somehow magically having these firms pay for this data.

Trust is earned, as is respect. Things like this both don't engender trust, and don't engender or show respect.

Maybe it's time for the company to take a step back - and ponder what went right in the past. Treating folks who've spent years contributing to the community well is likely to add value more than, well, trying to pick the most convenient narrative to quieten down dissent. Considering the result of actions such as this and the attempted gen AI policy does more than fixing the damage after the fact, and if these things are unforeseen or worse the fallout is seen but ignored, I feel like there needs to be a relook at community strategy.

Answer 8

I don't think that anything the community has written was confusing or conflicting. We realized the June data dump was missing, we asked for it, an external, very courageous source confirmed that the data dumps had been stopped. Then there came the official response that the data dumps have been switched on again and confirmation that the June data dump indeed run through. So far, so clear. The conflict about non-continued data dumps seemed to have been averted.

The only unclear thing is some vague talk about mysterious "guard rails" that the company is thinking about implementing, whatever that really means. Unfortunately this post here doesn't clarify the details behind that either. Maybe it was a bit premature and should have been posted later when more details are available.

As a reassurance it comes across rather weak. I wanted to hear something like "we are doing all we can to enable the free availability of the data dumps and other content access pipelines for now and all future." What I heard instead was some half-hearted commitment, some kind of "yes, but with restrictions for maybe some time". There is not much positive effect coming from that. My trust hasn't increased. I will wait if the September data dumps arrive in the usual format. Either it will or it won't. That will be my test.

Just two comments more:

Building additional services around the existing data offers and charging for this service is fine. If that's what you aim for, please say so.

The data isn't yours. It's a gift from all content creators to everyone (including you) in the world. We want it to be used as much as possible and that probably also includes your competitors. The license and the data dumps were supposed to be the means of that. Sure, there may be legal battles about the legality of using such material for AI training coming, but guard rails of the content aren't our concern really. We don't need them. This feature actually wasn't broken and to me the conclusion is that the company tinkering with it has a dis-proportionally high risk of making it worse. I'm very much afraid of what bad things you might have in mind there with something I value highly and this post didn't reassure me in any way.

Answer 9

The company is committed to the long-term (foreseeable future) survival of the data dumps, the API, and SEDE.

Will the license of the data dump change?

Answer 10

We retain the right to place guardrails around them to ensure that companies constructing language models, etc, are charged for access...

This use is allowed at no cost by CC BY-SA (although they do need to follow the license). Will SE try to interfere with legitimate, authorized usage of user content?

Answer 11

This is a problem:

We retain the right to place guardrails around them

You never really had that right, and asserting that you "retain" it suggests SE fails to grasp this key point. You can modulate access to the API - albeit at risk of community revolt if done unreasonably. You can't limit the distribution of the dumps. Not now, not ever.

This claim "sets conditions" for SE to attempt some form of access control in the future, and by your wording, the quarterly archive dumps are claimed to be fully within the scope of your discretion. I explained at length in this answer, but suffice to say they aren't. (Unless you give truly unlimited access to the same data by other means - which is probably a worse value proposition for SE in terms of bandwidth, resources, and future product development.)

It seems like you're really working hard to reestablish community trust. What we need is a specific re-commitment to maintaining the public data dumps, without any access control, ad infinitum. Ideally this should come from Mr. Chandrasekar himself, accompanied by an apology, as we are all well aware of his role in disabling them to begin with.

Your question (Interim) Policy on AI-content detection reports was a welcome surprise, and I am willing to extend some grace here. However, as a statement of policy, this response simply doesn't cut it. At a minimum, you need to revise this question to exclude the data dumps from your "reservations", and explicitly state that you will maintain them perpetually, in proportion with the perpetual use rights SE seeks for contributions.

You have everything to gain by such commitment, and - because of the practical and legal requirements of the CC license - absolutely nothing "real" to lose. As an aside, it's also something that the company already committed to in the past, so walking it back, even rhetorically, is a particularly bad look. Because your immediate actions are trending in the right direction, I'm willing to (provisionally) accept the strike agreement. But I'm not going to have much interest in making future contributions as long as this particular "threat" is dangling.

For a full resolution, we need SE to recognize at an organizational level that making the data dumps — publicly, entirely unencumbered by access control or financial expectations — is the "price" of our data. It's not something which you can "work around" or pay lip service to. It's a legal and moral obligation. It's also a fantastic deal for SE. Can you show us that you understand this?