16

Thanks to everyone who posted bug reports and feature requests related to the updated data dumps process. Below, we’ve detailed some work on those reports and requests.

Issues reported on this post:

  • Reverted to C# for producing XML to avoid illegal characters SQL Server misses.

  • ViewCount attribute removed from answers, instead of showing 0 (reported on this comment).

  • GUIDs are lower case to match SEDE (also reported on the previously linked comment).

Votes > type 16 are filtered out correctly (reported here).

Deleted posts are no longer referenced in certain places (reported here).

Changes to supplemental files:

  • A license.txt is present inside each .7z (reported here). These are present in regenerated files for the 2024-Q2 dumps, now available for download.

  • A sede-and-data-dump-schema.md is present inside each .7z, which is a snapshot of this answer's markdown at generation time. These are present in regenerated files for the 2024-Q2 dumps, now available for download.

  • A set of SHA256 checksums for the regenerated 2024-Q2 dumps has been posted in this answer. Future checksums can be published in whatever place makes the most sense, but cannot be distributed as part of the downloads.

  • The difference described here but not reported on Meta has been addressed as well: stackoverflow.com.7z now contains .xml files, not another level of .7z files.

We will not be taking action on the following:

  • Including date/timestamp of generation in file names (requested here). That metadata is already part of the file.

  • False positives from protection software that flag the .7z file but not the contents (reported here). This is an external issue that we cannot resolve.

This completes work for this stage of updates to this process. We’ll continue to review bugs and needs that arise and consider those when planning future updates. We do not have additional work planned before the next data dump (2024-Q3, to be posted in October as previously outlined).

Improve this question
23
  • 33
    Any update on the legal questions Philippe said on July 20 that we'd hear "early next week"? It's been more than a month. This is in regard to the legality of SE adding additional restrictions onto our content.
    – Jeff Bowman
    Commented Aug 29 at 17:27
  • 1
    @JeffBowman That update was provided in the July 26 update on that post.
    – Berthold StaffMod
    Commented Aug 29 at 17:29
  • 16
    What a shame. Do you think this is reasonable, and what the community expected when it contributed?
    – Jeff Bowman
    Commented Aug 29 at 17:36
  • 9
    Thanks for the fixes! I assumed any would need to wait until the next scheduled release, so it's nice to be getting them in an off-cycle update like this instead.
    – Jeremy
    Commented Aug 29 at 17:53
  • 10
    In spite of the many nay-sayers, and while I still don't think this new process is a very community-friendly approach (nor do I think it will accomplish what SE thinks it will), I do appreciate the time, effort, and thought that clearly went into addressing these issues (even if some of the results are, to some, imperfect). That anything has happened at all - never mind faster than 6-8 weeks - shows that there are still human beings in there somewhere...
    – testing-for-ya
    Commented Aug 29 at 20:35
  • 12
    @testing-for-ya The new dump method requires 920 clicks and plans to cut the data access to some users even if they follow the CC license, so yes, it is not a very community-friendly approach. The "nay-sayers" are just trying to save the community data. I'm guessing you've never publicly backed up the community data.
    – Franck Dernoncourt
    Commented Aug 29 at 23:26
  • 3
    @testing-for-ya "Just like (again, I'm sure) not many people would bother clicking 300+ times when the dump was only on archive.org" No need to click, one can download it via CLI. 1-line command: ia download stackexchange --retries==100. Many people downloaded the dump eg see meta.stackexchange.com/q/134495/178179
    – Franck Dernoncourt
    Commented Aug 30 at 0:11
  • 4
    @testing-for-ya let's not fool ourselves. The reason for this change is blatant. There is no ethical concern, no charity effort to protect us from evildoers that always used our data to train LLM and will continue to do so (just browse Hugging Face). The only point is that someone high up realized there is little money do be made in the few dying and abandoned "product" the company has and therefore is now desperate for other sources of income. [cont]
    – SPArcheon - on strike
    Commented Aug 30 at 9:38
  • 2
    @SPArcheon-onstrike Yeah, I'm not defending this change, or trying to make it less bad than it is. It's awful. I'm just trying to highlight the distinction between the people at Stack making all the bad decisions and the people at Stack that are still carrying out the work and trying to make a really bad situation slightly better.
    – testing-for-ya
    Commented Aug 30 at 11:36
  • 3
    @testing-for-ya oh, I wasn't blaming you for anything, sorry if I gave that impression. The point was just that the reason of the change is pretty obvious despite all the "we are doing this to protect your work from the evil no-attribution LLM". Saying that they are not required to post any dump and that we should appreciate the act of charity is also forgetting that at the same time we are not required to help their site, we are not required to moderate or curate the content and we are not even required to vote reasonably (see Tim's key lemma) [cont]
    – SPArcheon - on strike
    Commented Aug 30 at 11:54
  • 7
    @testing-for-ya when the network was created the founder had a social contract with the users to work toward a goal. Now the founder is gone and they are going back on that. And they can do that, there was no legal binding nor anything that keeps them in place. But they can't expect the users to be fine with it, they can't expect them to keep curating thing if they fell betrayed and they can't expect them to not whine about the unethical changes.
    – SPArcheon - on strike
    Commented Aug 30 at 11:55
  • 7
    And sadly, while obviously this is not Berthold fault as the one who picked the shortest straw and had to post this, the provide updates once again cherry picked some easily solvable points while missing the core ones. Just one simple example - some time ago someone pointed out a big fallacy in the data dump licensing detail that based on the wording of the CC license actually resulted in automatic termination. The issue was resolved, but the company carefully avoided even acknowledging the existence of the revocation clause. And this is only one of the "I don't hear you" issues.
    – SPArcheon - on strike
    Commented Aug 30 at 12:01
  • 3
    @AndreasmovedtoCodidact Please, please, please pay attention to context. I'm talking about the work done this week. I am absolutely not applauding anything that went into creating the mess we're in overall. And I'm not applauding "the company" in any way, shape, or form. I'm trying to separate the company from its soldiers, a distinction that is sometimes impossible to detect from some of the misery posted here.
    – testing-for-ya
    Commented Aug 30 at 17:00
  • 9
    @testing-for-ya Pardon, I’m just a bit tired of seeing the pattern. SE does something entirely stupid, then repairs the damage a tiny bit, to then await applause for that. All this is while they’ve purposefully not addressed the most important aspects, and aren’t going to. And I’m also worn down by having to separate the soldiers from the command. They are the company’s representatives, and they are the ones carrying out the commands. Small repairs like these matter none in the bigger picture, and I’m tired of seeing my digital friends here get burnt out at the actions of the company.
    – Andreas moved to Codidact
    Commented Aug 30 at 20:52
  • 3
    @testing-for-ya I can agree with your first comment under this post, but I cannot agree with the last response to SPArcheon; I've witnessed enough to know that it isn't so black-and-white as leadership vs pawns on the floor. "From the sounds of it, you'd rather they not fix anything than fix the things they can?" No, I want the company's attitude to stop misleading people here. I see so many friendly faces here waste so much time and effort trying to constructively interact with SE over and over again, and it gets them nowhere but burning out. I've burnt out myself, I don't want it to spread.
    – Andreas moved to Codidact
    Commented Aug 30 at 21:09

2 Answers 2

Reset to default
13

A sede-and-data-dump-schema.md is present inside each .7z, which is a snapshot of this answer's markdown at generation time. These are present in regenerated files for the 2024-Q2 dumps, now available for download.

Is there a commitment by the company to ensure that this is updated prior to a data dump being created? That is, if there is a change to the schema, will this be updated before the creation of the data dump to ensure that the data dump contains a complete and correct description of how to work with the data? If not, I'd consider this an incomplete item since it doesn't resolve the underlying issue.

One example of an error is that the markdown file says that EmailHash in the Users.xml is "now always blank". However, it's not always blank. I would interpret that as the attribute appearing and being an empty string. I don't see the attribute at all. So there is at least one error in the markdown file.

Future checksums can be published in whatever place makes the most sense, but cannot be distributed as part of the downloads.

Can the checksum be put onto the page where the download happens? This is, in my experience, a common practice. The download page would have the button to download the file along with that file's checksum (and any details needed to verify the checksum, such as the algorithm used).

A license.txt is present inside each .7z (reported here). These are present in regenerated files for the 2024-Q2 dumps, now available for download.

Is the license file correct?

The license file says that republishing the content requires certain elements of attribution. However, these cannot be levied as requirements.

Improve this answer
3
  • 3
    Regarding future checksum location, that is certainly one possibility. There are various ways that could be done. As noted in the answer on the checksum question, we'll wait to see what option(s) the community wants us to look at.
    – Berthold StaffMod
    Commented Aug 29 at 17:40
  • 9
    Yes, there is the commitment to ensure that sede-and-data-dump-schema.md is updated prior to a data dump being created. The markdown file is copied from the community-managed answer. So the community has the ability to correct any errors that might exist.
    – Berthold StaffMod
    Commented Aug 29 at 17:45
  • @Berthold Thank you for those clarifications. Just wanted to point out one last later addition question for clarification, if you're able to.
    – Thomas Owens
    Commented Aug 29 at 18:07
-5

The data dump access form states:

I understand that this file is being provided to me for my own use and for projects that do not include training a large language model (LLM), and that should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump.

but the license.txt present inside each .7z states:

All content contributed to Stack Exchange sites is licensed under the Creative Commons CC BY-SA license.

One of these two statements is incorrect, since those are incompatible statements. Can you please fix it?

Improve this answer
11
  • 21
    It's been mentioned in many places in recent weeks: production of the data dump is not a requirement, and they don't have to give it to everyone. They can decide you're not worthy if they catch you passing it on to an LLM company, or are getting sick of discussing animated GIFs, or they don't like your avatar. That has nothing to do with the license on the contents.
    – testing-for-ya
    Commented Aug 29 at 17:52
  • 6
    I agree with the first comment here, that there is no clash between the statements, and neither is really "incorrect". The situation might be unfair, perhaps considered unethical, some might try to prove it illegal, but technically, don't think it's incorrect.
    – Shadow Wizard
    Commented Aug 29 at 20:14
  • 2
    @ShadowWizard Right. We don't have to like it, but SE is absolutely free to give the data dump it produces to all, some, or none, for whatever reason(s) they feel like. They are also free to stop producing it altogether, which I fear is what will happen if they keep investing time and effort to make it better (amidst clearly bad strategy decisions) and are just met with more complaints and nit-picking. Then someone who is worried about what might happen when they check the legally unenforceable checkbox can get the data elsewhere because they will no longer have a choice.
    – testing-for-ya
    Commented Aug 29 at 20:29
  • 2
    @ShadowWizard "this file is being provided to me for my own use and for projects" vs. this file is under the Creative Commons CC BY-SA license.
    – Franck Dernoncourt
    Commented Aug 29 at 20:45
  • 6
    But "this file" is still given to you by choice and that license doesn't apply until they give it to you. They don't have to give it to you, or to anybody, if - for example - they think you're going to do (or have already done) something with it that they don't like. So they don't have to give you the next one in that case, and there isn't a thing CC will do for you about it. Once you have it, though, there's nothing at all they can about what you do with the contents.
    – testing-for-ya
    Commented Aug 29 at 20:59
  • 7
    @testing-for-ya then they should only say "should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump.". Because "this file is being provided to me for my own use and for projects" is incompatible with this file is under the Creative Commons CC BY-SA license.
    – Franck Dernoncourt
    Commented Aug 29 at 21:03
  • 5
    The checkbox is to get access to the file, not to use the file. You don't have to check the box after you've downloaded it, when you open it, or when you feed it to an LLM. Anyway several people have demonstrated that you don't have to check the checkbox and you are obviously free to get the files from any other source you like if you can't come to terms with the wording on the download page.
    – testing-for-ya
    Commented Aug 29 at 21:08
  • 3
    Aa I answered to one of your latest questions, this is not a bug that needs fixing. The checkbox agreement solely affects the distribution (SE can revoke access), beyond that, it doesn't have any legal significance nor any effects on how we can use the Data Dumps. That is and will be under CC-By-SA (unless PC is fine with getting SE sued).
    – M-- Save the Data Dump
    Commented Sep 2 at 22:35
  • Read the whole thing: "... should I distribute this file for the purpose of LLM training, Stack Overflow reserves the right to decline to allow me access to future downloads of this data dump." All they do, and can do, is that they would revoke your access to their instance of data dump. You could still access mirrors and use them under CC-By-SA without any consequences.
    – M-- Save the Data Dump
    Commented Sep 3 at 0:12
  • 1
    What they should say and their problematic agreement has been already discussed under the announcement post. One of the reasons you are getting downvoted is that you keep asking the same or a similar question over and over. The new process would only affect how you'd obtain (or not) the data dump from SE. Everything else is semantics.
    – M-- Save the Data Dump
    Commented Sep 3 at 0:18
  • @M--SavetheDataDump yes but OP totally ignored it, hence my reminder. Unlike you, and very likely, all downvoters, I reshare SE data publicly and train LLMs under my name, so this is of interest to me.
    – Franck Dernoncourt
    Commented Sep 3 at 0:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .