Questions tagged [data-dump]
This tag is about the quarterly creative commons data dumps of all public data in the Stack Exchange network Q&A sites.
516
questions
67
votes
2
answers
33k
views
Where are the Stack Exchange data dumps?
I see a lot of talk around here regarding the Stack Overflow public data dump.
Where is it? How can I play with it?
Return to FAQ index
- 22.6k
84
votes
3
answers
14k
views
All Stack Exchange data dump releases
I know how to download the latest release of the Stack Exchange Creative Commons Data Dump, but I'm interested in downloading a past release and don't see those listed. Where can I find them?
- 22.6k
10
votes
0
answers
154
views
What is a pre vote?
As seen in
New Vote Types in latest data dump?,
some new vote types appeared (likely) exclusively in the Stack Overflow data dump. A couple users helped find out what each one meant. However, what ...
- 14.7k
12
votes
1
answer
159
views
Checksums for data dumps should be included in the dump announcement and on the dump download page itself
Following on from my previous request for current checksums
I'm looking at 2 use cases for a data dump:
verifying if a current download for a data dump is correct
verifying if a specific historical ...
- 629
384
votes
2
answers
126k
views
Database schema documentation for the public data dump and SEDE
Stack Exchange releases "data dumps" of all its publicly available content roughly every three months via archive.org, and also makes that information queryable over the Internet at the ...
- 629
152
votes
4
answers
28k
views
Shifting the data dump schedule: A proposal
My colleague Rosie recently announced that, because of our difficulties predictably getting data dump files to the Internet Archive[1],[2], we are setting more realistic expectations for completing ...
CommunityBot
- 1
21
votes
2
answers
404
views
Could checksums be provided for the july 2024+/'new' SE hosted data dumps?
One of the nice things with the previous 'torrent' based downloads of the dump, and some of the internet archive front ends was that there were built in ways to verify that a download was correct.
...
- 173k
16
votes
2
answers
1k
views
Data Dumps - updates and bug fixes
Thanks to everyone who posted bug reports and feature requests related to the updated data dumps process. Below, we’ve detailed some work on those reports and requests.
Issues reported on this post:
...
- 1,877
16
votes
1
answer
245
views
The per-site data dump downloads are missing metadata files
When the data dumps were hosted on the Internet Archive, it was a torrent that consisted of the torrent file, a number of 7z files for each site, some images for branding, and two text files - readme....
- 1,877
16
votes
1
answer
440
views
My consent to the data-dump is not stored server side
When you ask me to consent to certain conditions like
I understand that this file is being provided to me for my own use and for projects that do not include training a large language model (LLM), ...
- 1,877
10
votes
1
answer
305
views
Last Data Dump has various objects that point to posts that do not exist
As I was working with the last SE data dump, I noticed that there are PostLinks that reference posts that do not exist.
As an example, in opensource.meta PostLinks.xml there is a row:
<row Id="...
- 1,877
12
votes
0
answers
312
views
One of my site data dumps appears to be giving a false positive for having a virus - what should I do?
I'm a moderator at Super User, and downloaded the dumps for that site. The 'main' dump's fine. The dump for meta failed because Windows detected it as having a virus. Since I got it from SE (and other ...
- 1,877
6
votes
0
answers
129
views
Individual data dump filenames should contain the dates they were generated
Historically, the data dumps hosted by clearbits or the internet archive were a single, monolithic 'set', and having a single datestamp for the collection made sense.
With the current system you're ...
- 1,877
21
votes
3
answers
466
views
Latest Data Dump has invalid XML and invalid characters
As I have been looking through the latest StackExchange data dump, it seems like a non-compliant XML serializer was used. There are numerous escape sequences that are simply invalid XML such as &#...
- 1,877
-377
votes
27
answers
33k
views
Announcing a change to the data-dump process
UPDATE August 29, 2024
This new post has updates related to various bug reports and feature requests.
UPDATE August 14, 2024
Over the past week we’ve scaled up the availability of the new page and ...
- 1,877
11
votes
2
answers
268
views
How to get SE sstatic images for my website
I've written about 2,500 questions and answers in Stack Exchange sites. About half of them I scrape into my website on GitHub Pages for customized presentation and searching.
A short time ago all the ...
9
votes
1
answer
231
views
Am I allowed to publicly reshare some JSON file containing SE data created after the introduction of the new data dump process?
I ran across some JSON (magnet link) containing SE data that was created after the introduction of the new data dump process. Am I allowed to publicly reshare it (e.g., on https://archive.org), or ...
- 2,284
10
votes
1
answer
374
views
How can Stack Exchange prove a violation to the Data Dump download agreement?
In the last few weeks there has been lot of talks about the new Data Dump process. From people asking if the dumps are watermarked to people pointing out that the user agreement for the download ...
- 43.8k
17
votes
1
answer
343
views
New Vote Types in latest data dump?
As I was looking through the latest StackExchange data dump, I noticed that in the StackOverflow dump there are a few new VoteTypeIds: 19,29,30,31,32,33
There is no mention of them in SEDE: https://...
- 22.6k
8
votes
2
answers
197
views
How can I get a notification when a new Stack Exchange dump is made available?
How can I get a notification when a new Stack Exchange dump is made available?
- 22.6k
14
votes
1
answer
906
views
Create a data dump RSS feed [duplicate]
Since the Stack Overflow data dump will occur monthy, it makes sense to be able to be notified about a new dump and download it. The current solution has been to post a link to the .torrent file on ...
- 22.6k
3
votes
1
answer
121
views
Can we please have some sort of API to poll for new data dump releases? [duplicate]
I'm writing an application that uses the data dump. However, in order to be effective, this application needs to be kept up to date with the latest data dump. This currently needs to be done manually.
...
- 22.6k
9
votes
1
answer
140
views
Can I get notified when there's an SE Data Explorer data dump? [duplicate]
As it is now, I'm checking every day if there is a new data dump. If I were notified, I would not have to keep checking.
- 22.6k
7
votes
0
answers
301
views
How can I get notified about released Stack Exchange dumps? [duplicate]
I am interested in being notified when a new data dump is published on the internet archive. I want to find some way to be automatically notified - ideally an RSS feed, but an email or something else ...
- 22.6k
3
votes
2
answers
217
views
Checksums for Sept 2017 data dump please
Is anyone facing issues with the data dump (web download) from archive.org? I've tried downloading the files multiple times over the last few days and I keep running into problems with the xml. (...
- 22.6k
32
votes
3
answers
376
views
Creative Commons License (BY-SA) Violation: Data dump must not force users to agree to additional terms
The new mechanism for downloading the Data Dump appears to be be live. It still has a checkbox that requires acknowledgement of "additional terms" in addition to those included in the ...
- 8,101
13
votes
1
answer
262
views
Are the new data dumps watermarked?
Are the new data dumps watermarked? I.e., can someone identify which user downloaded them?
- 54.1k
27
votes
2
answers
748
views
Last Data Dump is completely missing ContentLicense for Comments
I noticed that in the latest data dump, the Comments.xml files do not provide a ContentLicense for any of the rows, even though it is available in SEDE.
Here is an example of looking at the first few ...
- 22.3k
-2
votes
2
answers
477
views
Does SE plan to decline providing future versions of the data dumps to individuals who may use or share it for training LLMs?
The recent changes to data dump requires an agreement (emphasis is mine):
I understand that this file is being provided to me for my own use and for projects that do not include training a large ...
- 2,284
10
votes
2
answers
417
views
Do the new Data Dump terms of use allow commercial use NOT involving LLMs?
Announcing a change to the data-dump process says that the data dump download will be gated behind the following checkbox:
"I understand that this file is being provided to me for my own use and ...
- 2,284
30
votes
4
answers
904
views
Options for community data dump
We are in the process of automating the community data dump process and have reached the stage where we need to decide how and where to host the exported files.
We expect to have one data dump for all ...
- 11.6k
12
votes
0
answers
201
views
Opining on, and making a case for additional classes of data dumps
The data dumps traditionally were primarily meant as an insurance policy against the company going rogue.
Considering that the company 'now' sees the dumps as something they can turn into a product, ...
- 186k
26
votes
3
answers
538
views
Even without the checkbox, are SE's changes legal?
This relates to StackExchange's planned changes to the data dump process: Announcing a change to the data-dump process
One important point that I haven't seen anyone mention is that, in order to ...
- 8,314
46
votes
2
answers
1k
views
How can SE gate access to the Dump that will allow individuals access to the data while preventing "misuse" by for-profit organizations?
I read this answer written by Jody Bailey:
We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations ...
- 26.7k
35
votes
10
answers
2k
views
Why are people so unhappy about the OpenAI partnership? Am I missing something?
I'm reading the answers at Our Partnership with OpenAI, and a lot of people seem very unhappy about the partnership.
I feel violated, cheated upon, betrayed, and exploited.
And so on. But... why? ...
12
votes
1
answer
333
views
What are the indices used in SEDE?
I'm writing an application that uses the Stack Exchange Data Dump. I've imported the most recent dump into SQL Server 2014 on my local development machine.
I wrote a query that runs very slowly on my ...
CommunityBot
- 1
41
votes
1
answer
348
views
Why does the SEDE ReviewTaskResults table not include user data, which is all public anyway?
I wanted to do some data mining on the SEDE, and I ran across a limitation which I cannot understand.
The Data Explorer table ReviewTaskResults currently contains the following information:
Id
...
- 19.5k
68
votes
0
answers
9k
views
Data Dumps Releases: Timeline Updates and Clarification
We release our quarterly Data Dump towards the end of every quarter. We begin processing the Data Dump at the beginning of the last month of a quarter, but for the past three quarters, we’ve run into ...
- 3,575
82
votes
6
answers
2k
views
Dump of my own Stack Exchange content
Is there a way to quickly save all my Stack Exchange content across all Stack Exchange sites, as hundreds of HTML pages? (for me probably 15 or 20 sites: SO, dsp.SE, unix.SE, etc.)?
The ideal way ...
- 171
3
votes
2
answers
214
views
CC Data-dump: Are images included?
I am wondering whether images that are embedded into posts are included in the cc data dumps? If so, how?
- 43.8k
7
votes
1
answer
148
views
Periodically publish a data dump of all IDs in the image ID space (and make it available in SEDE)
I see that Imgur has an API to query for image IDs in its ID space, but api.stack.imgur.com is not an API endpoint (and SE is moving off Imgur anyway). Say I want to get all the IDs of images in Stack ...
- 27.5k
11
votes
1
answer
2k
views
How can I download the Stack Exchange data dump from archive.org through the command line?
I am trying to download the Stack Exchange data dump from archive.org through the command line using the internetarchive python package developed by archive.org, as follows:
pip install ...
- 43.8k
38
votes
1
answer
775
views
Delay in Data Dump release
Our quarterly publication of the Stack Exchange Network data (the “data dumps”) is currently delayed. While the process of copying and uploading data was scheduled to begin on Sunday, December 3rd, we’...
- 6,164
166
votes
31
answers
11k
views
Interesting queries on Data Explorer [closed]
Let's collect some interesting queries for the Stack Exchange Data Explorer here.
How many upvotes do I have for each tag? (how long before tag badges?)
SELECT TOP 20
TagName,
COUNT(*) AS ...
- 4,491
315
votes
13
answers
41k
views
June 2023 Data Dump is missing
The data dump usually gets refreshed the first weekend of the month, every 3 months.
The current data dump is still from March. Is there just a problem and it's delayed like in the past?
Relevant ...
- 4,491
1
vote
0
answers
154
views
Why is the 'Favorite' count so low in the Stack Overflow data? [duplicate]
In the Stack Overflow database, there is a column called "favorite count". I have found that favorite count indicates users find the question interesting or save it for later review.
I have ...
Aaron BertrandStaff
- 42.9k
72
votes
1
answer
1k
views
Possible delay in Data Dump release - now targeting September 5th
As you probably know, it’s time for the quarterly publication of our data (affectionately known as the “data dumps”). The process of copying, uploading, and publishing the data over a notoriously ...
- 9,326
97
votes
0
answers
697
views
Will SE shut down the free API for good? [duplicate]
SE has intentionally disabled the data dump. And the CEO has stated to the press that he plans to charge AI companies for access to the data.
Given these developments I think it is almost inevitable ...
CommunityBot
- 1
14
votes
2
answers
1k
views
Is SE in violation of the CC BY-SA 4.0 license by not providing data dumps?
SE is no longer providing data dumps for the content contributed by their users.
CC BY-SA 4.0 states:
No additional restrictions — You may not apply legal terms or technological measures that legally ...
- 14.9k
64
votes
11
answers
4k
views
The company's commitment to the data dumps, the API, and SEDE
Many words have been written around the company's commitment to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE). Much of that text can be confusing or ...
- 6,675