16

When the data dumps were hosted on the Internet Archive, it was a torrent that consisted of the torrent file, a number of 7z files for each site, some images for branding, and two text files - readme.txt and license.txt.

In this old format, the "unit of distribution" could be viewed as the network-wide data dump. You can choose which files to download, and obtain the license and readme if needed. In the new format, the torrent file and the images are no longer needed, but the readme and license are missing. This means that someone downloading or sharing the file doesn't have off-network access to the license the data dump is under or information about how to read it.

These two files should be embedded within each 7z file that is downloaded.

This is also a good opportunity to update the readme. I looked at the version included in the last data dump posted to the Internet Archive:

  • The TagBased attribute in Badges.xml isn't indicated as an optional element. It's not present for badges that aren't associated with a tag.
  • The ContentLicense attribute isn't mentioned as an attribute in the Comments.xml, PostHistory.xml, Posts.xml.
  • Users.xml no longer contains an EmailHash attribute.

These are just three examples of errors that I quickly found.

Since one purpose of the data dump is to allow people to continue to have access to and use the data in the event something happens to the network, relying on a network post to document the data dump doesn't make sense.

The lack of a license file means there is no information about the rights of the recipient of the data dump. The Terms of Service says that the data dump "is licensed under the CC BY-SA license", but it doesn't specify a version. Neither the download page nor the help center page for the dump provide a version, either. However, including the specific license on one of these pages doesn't help with resharing.

The license of the last version of the data dump uploaded to the Internet Archive was CC BY-SA 3.0. However, the text uses a legacy name, cc-wiki. This could be a good opportunity to license the data dump under CC BY-SA 4.0.

Alternatively, if the data dump is no longer licensed CC BY-SA, then the Terms of Service would need to be updated. Although I don't fully understand the value in licensing it as anything else, given that the vast majority of the content in the data dump is under one of three CC BY-SA licenses.

If possible, the wording around the attribution requirements should be revisited, keeping in mind that licensors "may make special requests, such as asking that all changes be marked or described", but these requests are non-binding, even though CC asks licensees to consider complying with these requests. However, item 3 in the attribution request ("show the author names for every question and answer") is a hard requirement in some cases, when a large enough quantity of a specific post is reproduced or transformed.

Improve this question
6
  • Which site(s) has the new dumps? I don't see it on my side when going to my profile -> Settings -> Access.
    – Franck Dernoncourt
    Commented Aug 8 at 11:03
  • 6
    @FranckDernoncourt It's being rolled out in batches. Guess you're just unlucky, and will have to wait a bit longer.
    – Tinkeringbell Mod
    Commented Aug 8 at 11:06
  • Is the concern about the missing licensing file somewhat ameliorated by the ContentLicense comment with license URLs that has been added? I think it would definitely be better to have a clearly written license file included, but I was thinking the comment might at least satisfy a minimum requirement.
    – Jeremy
    Commented Aug 11 at 21:57
  • 1
    @Jeremy No. ContentLicense applies to the content submitted by users. It doesn't describe the license of the collection. Without a license on the data dump as a collective work, a downstream recipient wouldn't know their rights as it pertains to the data dump as a whole.
    – Thomas Owens
    Commented Aug 11 at 22:01
  • But previously, anyone could download any single file from archive.org. If they didn't also download the license.txt file (which is still there by the way) and pass it along, wouldn't a downstream recipient still be in the same boat of not knowing their rights as it pertains to the data dump as a whole? Is there any reason to assume the license has changed from the version that is still available on archive org? Is there any reason it was okay that a license.txt file wasn't previously jammed into every .7z file?
    – user1502910
    Commented Aug 11 at 22:14
  • 1
    @testing-for-ya I mentioned this in my question. What was uploaded to IA was the collective work - a set of per-site 7z files, some images, a readme, and a license file. Sure, someone could opt to not download the license file, and it would be the same situation as now if someone tried to redistribute without a license. However, there's no now longer a canonical source with a license file that exists. The only place the mentions a license is the terms of service, and it doesn't mention a version of the license - saying it's CC BY-SA is insufficient without saying which CC BY-SA.
    – Thomas Owens
    Commented Aug 11 at 22:39

1 Answer 1

Reset to default
5

This change has been made. Please see this post for more details on other fixes and changes related to data dumps.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .