14

SE is no longer providing data dumps for the content contributed by their users.

CC BY-SA 4.0 states:

No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

By not providing the data dumps, it would make it necessary for anyone wishing to use the data to essentially parse the HTML on the site, instead of, e.g. directly using the data dump. This very much sounds like a violation of the "technological measures" clause above.

So, is SE in violation of the CC BY-SA 4.0 license?

3
  • 10
    No. Even their own ToS says that they only "may" publish these occasionally. Do you think they are also in violation for not publishing them daily or weekly? How about in braille, or text-to-speech, or translating to 47 other languages? They are not in violation of CC BY-SA 4.0 by not publishing anything in an alternative format. They also wouldn't be in violation by halting updates to data explorer or taking that interface away altogether. Now, are they being short-sighted, pissing off everyone while violating logic & basic sniff tests? Absolutely. But not for not removing HTML tags for you. Jun 12, 2023 at 11:32
  • 10
    It would be terrible for the CC-BY-SA license to do this, because it would mean that you would also have to provide a dump of any CC-BY-SA content you've used for any purpose, which makes it a whole lot harder to use CC-BY-SA content when the whole point of the license is to make it easier for others to use. Jun 12, 2023 at 14:33
  • 1
    See also now meta.stackexchange.com/questions/390043/…
    – tripleee
    Jun 13, 2023 at 4:42

2 Answers 2

49

I am not a lawyer, but I'm pretty much 100% sure Stack Exchange is not required to provide data dumps except that they promised the community they would do so way back when. CC BY-SA just describes the terms that a third-party can use the data if/when they get a copy of it, but it doesn't in any way specify that Stack Exchange needs to distribute that data in any particular form or at any particular convenience, or even at all.

If they choose to hide the data dumps behind a paywall or eliminate them entirely that's still valid, but under CC BY-SA they are not allowed to add additional restrictions to how a third party uses that data once they acquire it, nor can they prevent anyone from distributing that exact same data in a more convenient manner.

As for the "technical measures" clause, from the CC BY-SA 4.0 license:

Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.

I very much doubt "it's slightly more difficult to parse as HTML" counts under that definition.

That said, there's also still the whole dual-license thing to consider, but again IANAL and that goes beyond the scope of this particular question.

5
  • 8
    Parsing the HTML is inconvenient, but not so much of a problem; the real hurdle is that it is not possible to download the content page by page in bulk quantities, as you will quite soon hit a rate limit. Jun 12, 2023 at 10:59
  • 6
    Those promises are not legally binding, I'd imagine.
    – Richard
    Jun 12, 2023 at 11:56
  • 6
    The "effective technological measures" clause refers to DRM, i.e. copy-protection mechanisms which, under laws like the DMCA, it is illegal to break or circumvent. Parsing HTML is not illegal (yet), even if you do it with a regex.
    – kaya3
    Jun 12, 2023 at 15:39
  • A "rate limit" counts as an "effective technological measure" to the extent they make any effort to prevent (including in the ToS) distributed scraping. For all intents and purposes, the "WIPO" reference means DMCA. And because DMCA's vague and overbroad "anti-circumvention" provisions have been applied so ubiquitously to attack basically anyone who "bypasses a logon", almost any scraping which violates the ToS is intrinsically a DMCA violation. Precisely because copyright law is so regularly and egregiously abused is why almost anything SE does to limit access could count against them.
    – BryKKan
    Aug 21, 2023 at 11:13
  • (continued) In terms of the CC-BY-SA license, they are required to "share alike" the entire "adapted work(s)" that the dumps contain, freely, for any purpose, and without interference or further restriction. If they fail to do so, they jeopardize their own license to the entire Q&A site data (with a 30 day warning to fix it). So while it's technically true that they don't have to share the dumps, doing so is their "get out of jail free card" for using "technological measures" to protect the live site from the resource drain of mass scraping. Otherwise, they have to let us get it all directly
    – BryKKan
    Aug 21, 2023 at 11:19
1

So... yes and no.


First, the "No"

Technically most of the other answers saying "no" are correct, if incomplete. In the absolute strictest sense, they are not obligated to provide data dumps ever again, at least not by the license terms¹. That's because it doesn't require sharing anything we submit - the license gives them the right to stop using licensed content at any time. They also aren't specifically required by the license to collate or upload a full dump, much less to do so in some particular format.


Now the (qualified) "Yes".

A commonly missed or misunderstood part of the CC-BY-SA license is the "ShareAlike" provision. SE picked this license "flavor" because the SA version allows them to "remix" submissions freely, using them pretty much any way they want. But this freedom comes with a catch: they have to give others the same flexibility in turn.

Even derivative, or "adapted materials" must be shared under the same CC-BY-SA license. In fact, by the terms of the license, it is extended to all recipients, even if SE says otherwise! Regardless of whether they explicitly grant the CC license to 3rd parties, or specifically offer the data under another license: any recipient of the CC licensed content, even if only in the form of "adapted materials", still also gets an unrestricted and royalty-free CC-BY-SA license, just by virtue of receipt. If SE doesn't inform the recipient of the original CC-BY-SA license, they are in violation of the attribution section of the terms. If they do, then the 3rd party recipient doesn't need SE's permission to claim or use the CC license themselves. They simply have it, automatically.

As an aside, the license also prohibits SE from charging any kind of "royalty" for access to the content itself, which would arguably include most efforts to fully "guardrail" commercial data access. They can charge for other things besides access to the content, for instance preparing the data in a special format that's more portable, or allowing someone to run larger queries against the live database. But this is true only if they share the same data freely in some other manner. Otherwise any "monetization" plans can be construed as data access charges, thus violating the license.

Moreover, as long as they are sharing the licensed content at all, they are actually supposed to keep a URL link to it, along with notation of edits. There is some flexibility in the wording, and it's not a strictly universal requirement, but the exception is intended for data domains where the content is fully available in other forms, and where creating such links would impose an excessive burden. It would be a hard sell for SE to claim providing a link is infeasible, given the nature of the site.


How does this impose any requirement to do the data dumps?

Well, just because they (must) provide the content links doesn't mean they want everyone to use them exclusively. That's highly resource intensive, and unnecessary for compliance. They don't have to serve up a whole site page to satisfy the license, they just have to offer a URL that responds with the appropriate content and metadata. As an example, certain API functionality can substitute entirely².

However, they can't add any access control mechanisms which effectively prevent someone from retrieving a "live" piece of licensed content from the designated URL for that content, even if it's the 4723rd one they've asked for today. Someone may need to see that particular post for reference reasons. If the regular HTTP site is the only means of accessing the content, then it must be served to everyone. There is no way for SE to distinguish between allowable and disallowable reasons (if any even exist) for requests. A page load is a page load.

Again, they can take down content entirely at any time. But if they reshare in any form, in any context, they'd have to put the public link back up too. So it's not like they can keep a cache of "hidden gems", deleted in public, and stashed away for the paid version only.

The data dumps are simply the cheapest, most efficient, and most practical way for them to comply with the license terms. If SE continues to do full data dumps, then they have some leeway to rate-limit or otherwise "gate" the HTTP site, and to some extent even the API. If not, then they have to leave the API open, or let us scrape the HTTP site ad nauseam, regardless of performance impacts.


It's really more of a "yes, if..."

The dumps are only an obligation to the extent that other substitute means of accessing the data are hobbled by disallowed restrictions. There is no intrinsic requirement to maintain them, but if SE executes its intended plans to limit commercial access elsewhere, they will become required, as the only remaining license-compliant source for the content. In fact, depending on the nature of those other guardrails, they might even be required to increase the frequency of dumps, just to be safe.

All this is to say, they can opt out of the data dumps specifically, but probably not while simultaneously limiting HTTP requests - at least not without providing an alternative. The data dumps are their "get out of jail free" card. If someone complains about rate limits on the HTTP site, or certain restrictions on the API, they can be referred to the dump. And if someone needs to validate the dump, then minimal API/SEDE lookups - or at worst some light scraping - should usually suffice. More broadly, publishing the dump creates conditions in which they can fairly assert certain desired "guardrails" on other data channels without running afoul of the license terms.


Did they actually violate the license this time?

In the present case, the question of whether SE violated the license is moot, because they rectified the issue within the 30-day grace period allowed by the license terms. However, we can still examine it academically. All we need ask is: was the data ever unavailable without disallowed restrictions? Obviously, the HTTP site currently has impermissible "technological measures" which limit access, and thus cannot satisfy the license. So then it hinges upon whether API access was restricted at any point in early June (I don't know if it was). If they didn't lock down the API at all during that time, then the licensed content always remained fully available (at a URL, no less). In that case, they didn't violate the license. Otherwise - they probably did.


¹There is an argument to be made that past promises to always share dumps created an implicit contract, and that content created/shared before a public announcement ending them was only licensed under CC-BY-SA contingent upon fulfillment of that contract. That potentially justifies an argument for voiding the ToS contract which granted the license, and thus a "backdoor" means of rescinding the (otherwise perpetual) CC-BY-SA license even if they don't violate it directly. However, they could avoid that issue simply by announcing (with sufficient notice) that their last dump will be the last dump.

²Note that they don't have to offer the API either. It's just that offering open API access likely satisfies the license, thus freeing them to restrict or limit other forms of access.

22
  • How does "royalty" equate to data-access charges? Royalties are paid out for the continued usage of a work, and have nothing to do with how it's accessed. CC BY-SA very much allows profiting off of the work, otherwise there'd be no point in CC BY-NC-SA also existing.
    – goldPseudo
    Aug 22, 2023 at 3:40
  • @goldPseudo I was never suggesting they can't make a profit, but I appreciate the feedback. This statement is not very well explained, and it could certainly stand improvement. It's a bit complicated to show, and I didn't want to get further bogged down in an already long answer.
    – BryKKan
    Aug 22, 2023 at 4:09
  • (continued 1) To elaborate, the reason can't charge for data access generally has more to do with the nature of contributions to SE than the license itself. Because the licensed content is targeted to SE, and not publicly available elsewhere, the only way for any downstream recipients to fulfill the license requirements of attribution and source linking is to refer to the SE post, or a properly attributed archive thereof. Maybe they can offload that responsibility in some way other than public data dumps, but it doesn't really matter, because that's not what they're trying to do anway.
    – BryKKan
    Aug 22, 2023 at 4:23
  • 1
    I also don't see anything in the license obligating anyone to maintain an always-online server with a complete history of any modifications made over the entirety of the licensed work. It only requires inclusion of an URL as part of the attribution requirements, and even then only if provided by the licensor, which they have the discretion to remove at any time.
    – goldPseudo
    Aug 22, 2023 at 4:23
  • (Continued 2) The problem is that every single downstream recipient needs to be able to reference the source, and SE neither does, nor can have any control over who those recipients are. It could be literally anyone. Some college kid from China could be linking from a blog post, quoting a news article, reporting on some LLM. If she receives a copy of adapted adapted adapted material, she's still fully entitled to exercise the CC-BY-SA license of each adapted version. To do that, she needs to be able to attribute it, and have at least some means of pointing people to these versions.
    – BryKKan
    Aug 22, 2023 at 4:30
  • Yup, you're on the right track there. Sorry for the confusing crossover, I was alread writing the second message. They happened to dovetail well though, so I guess it worked out.
    – BryKKan
    Aug 22, 2023 at 4:31
  • (Continued 3) SE isn't required to maintain the content online, or even themselves. The archive dumps are obviously a much better suited reference anyway. The current 3-month interval means that some transiently published posts may be "missed" or intentionally skipped by the archive job. They might still have been referenced while live, and thus a license issue would arise downstream. But this would be rare, and the CC license doesn't require perfection. I don't see such edge cases being treated as license violations. On the other hand, if they don't make the dumps at all, they're in a pickle.
    – BryKKan
    Aug 22, 2023 at 4:41
  • Without data dumps to substitute as a reference, any deletions SE makes from the site effectively block downstream licensees from exercising their full rights in the content they received. Even if the the content and edit history remain live on the site, any mechanisms which limit access to the reference content potentially infringe upon downstream licensees. You have no way of knowing whether they received a copy of the reference material OR an adadpted material - including adapted material created downstream. It may barely even resemble the source. They still have a license to the source.
    – BryKKan
    Aug 22, 2023 at 4:51
  • Finally, note that if they charge money for such data access, they are effectively requiring a fee to these anonymous downsteam licensees, as a prerequisite to the full exercise of their rights. The whole arrangement starts to look a lot like a royalty. A critical point is that there is no valid measure of scale here. If I collate 200k SE submissions from 3rd parties, I can modify them in bulk and pass them on. But if somewhere down the chain, someone wants to see the originals, they've every right to want to grab them in bulk. And there's no way for SE to distinguish it from a bulk dump.
    – BryKKan
    Aug 22, 2023 at 5:01
  • 1
    Attribution and access are completely different things. ShareAlike means you need to attribute the source, not that you are obligated to ensure that the source is available at all time for all downstream users at no cost forever. Otherwise you, personally, would also be obligated to provide the same access to everyone if you, personally, ever chose to exercise your own right to adapt and ShareAlike, regardless of the status of the StackExchange database. I very much doubt this is how CC BY-SA was ever meant to be interpreted or applied.
    – goldPseudo
    Aug 22, 2023 at 5:09
  • @goldPseudo Yes, again you are right. They are quite distinct. However, in order to attribute any licensed or adapted materials properly themselves, SE needs to provide a URL to the licensed content, to the extent reasonably practicable. As the hoster and sole controller of the original source, it would be inane for SE to argue it's "not practicable". As long as SE is hosting it, downstream licensees can reference that. And stale data would be grandfathered if it went dark after a while. But it's reasonable to assume that if the SE attribution is impermanent, people will scrape while live.
    – BryKKan
    Aug 22, 2023 at 5:27
  • Any actively maintained adaptations or redistributions would be regularly hitting those source links to archive them for themselves in case they went down later, and SE can't use technological measures to prevent them from doing that independent archival as long as the licensed material is hosted. If SE sells data access, they have to provide valid links to all licensed content in the dataset. If they set an expectation that bulk data buyers can't use the links, they're infringing attribution rights. And if any downstream recipients do use the links, they're fully entitled to archive them.
    – BryKKan
    Aug 22, 2023 at 5:35
  • 1
    Isn't the API restricted, too? I know SmokeDetector was running into API quota issues.
    – tripleee
    Aug 22, 2023 at 19:04
  • 4
    Your argument here is premised on the idea that people who view content you've distributed have a right to view the original source. The license makes no such claim. It requires you to indicate if it has been modified; you don't even need to specify what was modified. The license does not in any way require the original to be available in order to continue using it, if you already had it. It only requires you to retain the link to where you found it.
    – animuson StaffMod
    Aug 23, 2023 at 0:34
  • 4
    If that link is dead, that has no bearing whatsoever on your use, only that it will be harder to prove you had the license to use it without contacting the party that originally hosted the content (assuming it's not in the Wayback Machine or something). But the link you retain as reference is not required to remain online. You are spending a lot of time in your answer expanding the license with wording that does not exist, and it makes your entire answer severely flawed.
    – animuson StaffMod
    Aug 23, 2023 at 0:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .