Questions tagged [data-dump]
This tag is about the quarterly creative commons data dumps of all public data in the Stack Exchange network Q&A sites.
516
questions
10
votes
0
answers
171
views
What is a pre vote?
As seen in
New Vote Types in latest data dump?,
some new vote types appeared (likely) exclusively in the Stack Overflow data dump. A couple users helped find out what each one meant. However, what ...
- 661
12
votes
1
answer
169
views
Checksums for data dumps should be included in the dump announcement and on the dump download page itself
Following on from my previous request for current checksums
I'm looking at 2 use cases for a data dump:
verifying if a current download for a data dump is correct
verifying if a specific historical ...
- 187k
16
votes
2
answers
1k
views
Data Dumps - updates and bug fixes
Thanks to everyone who posted bug reports and feature requests related to the updated data dumps process. Below, we’ve detailed some work on those reports and requests.
Issues reported on this post:
...
- 1,877
9
votes
1
answer
239
views
Am I allowed to publicly reshare some JSON file containing SE data created after the introduction of the new data dump process?
I ran across some JSON (magnet link) containing SE data that was created after the introduction of the new data dump process. Am I allowed to publicly reshare it (e.g., on https://archive.org), or ...
- 44.5k
21
votes
3
answers
482
views
Latest Data Dump has invalid XML and invalid characters
As I have been looking through the latest StackExchange data dump, it seems like a non-compliant XML serializer was used. There are numerous escape sequences that are simply invalid XML such as &#...
- 872
17
votes
1
answer
348
views
New Vote Types in latest data dump?
As I was looking through the latest StackExchange data dump, I noticed that in the StackOverflow dump there are a few new VoteTypeIds: 19,29,30,31,32,33
There is no mention of them in SEDE: https://...
- 872
6
votes
0
answers
130
views
Individual data dump filenames should contain the dates they were generated
Historically, the data dumps hosted by clearbits or the internet archive were a single, monolithic 'set', and having a single datestamp for the collection made sense.
With the current system you're ...
- 187k
10
votes
1
answer
375
views
How can Stack Exchange prove a violation to the Data Dump download agreement?
In the last few weeks there has been lot of talks about the new Data Dump process. From people asking if the dumps are watermarked to people pointing out that the user agreement for the download ...
- 32.7k
16
votes
1
answer
446
views
My consent to the data-dump is not stored server side
When you ask me to consent to certain conditions like
I understand that this file is being provided to me for my own use and for projects that do not include training a large language model (LLM), ...
reneMod
- 91.5k
13
votes
1
answer
262
views
Are the new data dumps watermarked?
Are the new data dumps watermarked? I.e., can someone identify which user downloaded them?
- 44.5k
32
votes
3
answers
381
views
Creative Commons License (BY-SA) Violation: Data dump must not force users to agree to additional terms
The new mechanism for downloading the Data Dump appears to be be live. It still has a checkbox that requires acknowledgement of "additional terms" in addition to those included in the ...
- 13.9k
12
votes
0
answers
319
views
One of my site data dumps appears to be giving a false positive for having a virus - what should I do?
I'm a moderator at Super User, and downloaded the dumps for that site. The 'main' dump's fine. The dump for meta failed because Windows detected it as having a virus. Since I got it from SE (and other ...
- 187k
21
votes
2
answers
411
views
Could checksums be provided for the july 2024+/'new' SE hosted data dumps?
One of the nice things with the previous 'torrent' based downloads of the dump, and some of the internet archive front ends was that there were built in ways to verify that a download was correct.
...
- 187k
16
votes
1
answer
251
views
The per-site data dump downloads are missing metadata files
When the data dumps were hosted on the Internet Archive, it was a torrent that consisted of the torrent file, a number of 7z files for each site, some images for branding, and two text files - readme....
- 54.1k
11
votes
2
answers
268
views
How to get SE sstatic images for my website
I've written about 2,500 questions and answers in Stack Exchange sites. About half of them I scrape into my website on GitHub Pages for customized presentation and searching.
A short time ago all the ...
- 459
-2
votes
2
answers
480
views
Does SE plan to decline providing future versions of the data dumps to individuals who may use or share it for training LLMs?
The recent changes to data dump requires an agreement (emphasis is mine):
I understand that this file is being provided to me for my own use and for projects that do not include training a large ...
- 44.5k
27
votes
2
answers
754
views
Last Data Dump is completely missing ContentLicense for Comments
I noticed that in the latest data dump, the Comments.xml files do not provide a ContentLicense for any of the rows, even though it is available in SEDE.
Here is an example of looking at the first few ...
- 872
10
votes
2
answers
419
views
Do the new Data Dump terms of use allow commercial use NOT involving LLMs?
Announcing a change to the data-dump process says that the data dump download will be gated behind the following checkbox:
"I understand that this file is being provided to me for my own use and ...
- 5,570
12
votes
0
answers
201
views
Opining on, and making a case for additional classes of data dumps
The data dumps traditionally were primarily meant as an insurance policy against the company going rogue.
Considering that the company 'now' sees the dumps as something they can turn into a product, ...
- 187k
10
votes
1
answer
307
views
Last Data Dump has various objects that point to posts that do not exist
As I was working with the last SE data dump, I noticed that there are PostLinks that reference posts that do not exist.
As an example, in opensource.meta PostLinks.xml there is a row:
<row Id="...
- 872
26
votes
3
answers
540
views
Even without the checkbox, are SE's changes legal?
This relates to StackExchange's planned changes to the data dump process: Announcing a change to the data-dump process
One important point that I haven't seen anyone mention is that, in order to ...
- 381
-382
votes
27
answers
34k
views
Announcing a change to the data-dump process
UPDATE August 29, 2024
This new post has updates related to various bug reports and feature requests.
UPDATE August 14, 2024
Over the past week we’ve scaled up the availability of the new page and ...
- 21.2k
35
votes
10
answers
2k
views
Why are people so unhappy about the OpenAI partnership? Am I missing something?
I'm reading the answers at Our Partnership with OpenAI, and a lot of people seem very unhappy about the partnership.
I feel violated, cheated upon, betrayed, and exploited.
And so on. But... why? ...
- 5,570
152
votes
4
answers
28k
views
Shifting the data dump schedule: A proposal
My colleague Rosie recently announced that, because of our difficulties predictably getting data dump files to the Internet Archive[1],[2], we are setting more realistic expectations for completing ...
Aaron BertrandStaff
- 42.9k
68
votes
0
answers
9k
views
Data Dumps Releases: Timeline Updates and Clarification
We release our quarterly Data Dump towards the end of every quarter. We begin processing the Data Dump at the beginning of the last month of a quarter, but for the past three quarters, we’ve run into ...
- 13.8k
7
votes
1
answer
148
views
Periodically publish a data dump of all IDs in the image ID space (and make it available in SEDE)
I see that Imgur has an API to query for image IDs in its ID space, but api.stack.imgur.com is not an API endpoint (and SE is moving off Imgur anyway). Say I want to get all the IDs of images in Stack ...
- 27.9k
38
votes
1
answer
776
views
Delay in Data Dump release
Our quarterly publication of the Stack Exchange Network data (the “data dumps”) is currently delayed. While the process of copying and uploading data was scheduled to begin on Sunday, December 3rd, we’...
- 13.8k
1
vote
0
answers
154
views
Why is the 'Favorite' count so low in the Stack Overflow data? [duplicate]
In the Stack Overflow database, there is a column called "favorite count". I have found that favorite count indicates users find the question interesting or save it for later review.
I have ...
7
votes
0
answers
302
views
How can I get notified about released Stack Exchange dumps? [duplicate]
I am interested in being notified when a new data dump is published on the internet archive. I want to find some way to be automatically notified - ideally an RSS feed, but an email or something else ...
- 13.6k
72
votes
1
answer
1k
views
Possible delay in Data Dump release - now targeting September 5th
As you probably know, it’s time for the quarterly publication of our data (affectionately known as the “data dumps”). The process of copying, uploading, and publishing the data over a notoriously ...
Jody BaileyStaff
- 807
3
votes
1
answer
123
views
Discrepancy in Post Counts between Local MySQL Stack Overflow Database and Data Stack Exchange Query
I recently downloaded the Stack Overflow data from Stack Overflow datadump and imported it into my local MySQL database. While attempting to retrieve the total number of posts for the year 2021 using ...
64
votes
11
answers
4k
views
The company's commitment to the data dumps, the API, and SEDE
Many words have been written around the company's commitment to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE). Much of that text can be confusing or ...
- 21.2k
3
votes
1
answer
591
views
How can I download a small data set (1 year 2023-2022) from Stack Exchange?
I want to download a small dataset for one year. I didn't find a link that allows me to download 1 year of data or the last five years' data.
I tried "https://data.stackexchange.com/stackoverflow/...
-9
votes
1
answer
262
views
Does Stack Exchange Inc. intend to sue firms that use language models that are partly trained on Stack Exchange?
Stack Exchange Inc. has expressed an interest in financially profiting from some companies that use Stack Exchange data to train their model.
Many language models are partly trained on Stack Exchange ...
- 44.5k
5
votes
1
answer
173
views
Why are the profileimageurl in the data dump of almost all Stack Exchange websites blank?
I wonder why the profileimageurl in the data dump of almost all Stack Exchange websites are blank.
2
votes
2
answers
198
views
Does Stack Exchange offer any incentives related to use of their data explorer in research?
I'm interested in whether or not there have historically been this sort of partnership with research in the academic space or if it is planned for the future. This could be something like a grant, ...
-10
votes
0
answers
1k
views
What is the current (June 2023) status of the data dumps and the company’s commitment to them? [duplicate]
What is the current (June 2023) status of the data dumps, the API, and the Data Explorer? Will the company continue to provide them going forward?
- 21.2k
14
votes
2
answers
1k
views
Is SE in violation of the CC BY-SA 4.0 license by not providing data dumps?
SE is no longer providing data dumps for the content contributed by their users.
CC BY-SA 4.0 states:
No additional restrictions — You may not apply legal terms or technological measures that legally ...
- 1,040
36
votes
0
answers
281
views
Is the 'restrictions'/'guardrails' on dump access going to affect the Overflow Offline project?
I've generally been interested in the CSR aspect of Stack Overflow Inc. One of the initiatives that Stack Overflow Inc. was working on was the Overflow Offline project.
As per your own media - it ...
- 187k
40
votes
0
answers
228
views
Stopping data dumps to "protect data" - harmful and threatening to the community
I find it a bit odd that "senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs", but historically has only cared about ...
- 54.1k
11
votes
0
answers
240
views
What initiatives exist to create a publicly available dump of SE under the CC BY-SA 4.0 license?
SE is not providing a dump for the time being. They may do it again in the future, but with some restrictive ToS/license/etc.
This makes me wonder: What initiatives exist to create a publicly ...
- 44.5k
46
votes
2
answers
1k
views
How can SE gate access to the Dump that will allow individuals access to the data while preventing "misuse" by for-profit organizations?
I read this answer written by Jody Bailey:
We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations ...
- 44.5k
3
votes
0
answers
213
views
When does the SE "senior leadership" plan to re-enable the uploads of the data dump to Archive.org? [duplicate]
I read this answer written by AMtwo:
The job that uploads the data dump to Archive.org was disabled on 28 March, and marked to not be re-enabled without approval of senior leadership.
This is a huge ...
- 44.5k
26
votes
2
answers
484
views
Could someone legally replace Stack Exchange in posting the data dumps to the Internet Archive?
Recently the user discovered that without any official communication the company stopped posting the monthly data dump to the Internet Archive web site. Causes are currently unknown but some users ...
- 32.7k
97
votes
0
answers
697
views
Will SE shut down the free API for good? [duplicate]
SE has intentionally disabled the data dump. And the CEO has stated to the press that he plans to charge AI companies for access to the data.
Given these developments I think it is almost inevitable ...
- 188k
316
votes
13
answers
41k
views
June 2023 Data Dump is missing
The data dump usually gets refreshed the first weekend of the month, every 3 months.
The current data dump is still from March. Is there just a problem and it's delayed like in the past?
Relevant ...
- 1,881
1
vote
0
answers
31
views
Where is the GitHub link in the Data Exchange Explorer located? [duplicate]
I went closely over the SEDE's fields, but couldn't find where the GitHub link is saved for a user:
I ran this query on my own profile but couldn't find my GitHub account.
- 383
0
votes
0
answers
61
views
Why isn't the summation of a user's score equal to their reputation? [duplicate]
I was going over the database schema documentation and was trying to understand how the Post scoring system works. I was expecting this query:
select sum(p.score)
from posts p
where p.OwnerUserId = ...
- 383
4
votes
1
answer
65
views
What is the difference between DisplayName and Id on the Stack Exchange Data Explorer Users table?
I am trying to pull a list of all users who have commented on an answer and tie it back to their User profile, if one exists. However, not all Comments table records have a UserId associated with them ...
13
votes
1
answer
200
views
Update the readme.txt of the data dump ... again
In 2017 I asked to update the readme.txt of the data dump.
Today I revisited that read me due to Stack Exchange Data Dump — OwnerDisplayName and LastEditorDisplayName. I was surprised to find that the ...
reneMod
- 91.5k