Skip to main content

Questions tagged [data-dump]

This tag is about the quarterly creative commons data dumps of all public data in the Stack Exchange network Q&A sites.

Filter by
Sorted by
Tagged with
30 votes
7 answers
1k views

Why are people so unhappy about the OpenAI partnership? Am I missing something?

I'm reading the answers at Our Partnership with OpenAI, and a lot of people seem very unhappy about the partnership. I feel violated, cheated upon, betrayed, and exploited. And so on. But... why? ...
HolyBlackCat's user avatar
  • 5,160
150 votes
4 answers
27k views

Shifting the data dump schedule: A proposal

My colleague Rosie recently announced that, because of our difficulties predictably getting data dump files to the Internet Archive[1],[2], we are setting more realistic expectations for completing ...
Aaron Bertrand's user avatar
  • 42.7k
68 votes
0 answers
8k views

Data Dumps Releases: Timeline Updates and Clarification

We release our quarterly Data Dump towards the end of every quarter. We begin processing the Data Dump at the beginning of the last month of a quarter, but for the past three quarters, we’ve run into ...
Rosie's user avatar
  • 14k
6 votes
1 answer
140 views

Periodically publish a data dump of all IDs in the image ID space (and make it available in SEDE)

I see that Imgur has an API to query for image IDs in its ID space, but api.stack.imgur.com is not an API endpoint (and SE is moving off Imgur anyway). Say I want to get all the IDs of images in Stack ...
super-starball-ultra's user avatar
38 votes
1 answer
727 views

Delay in Data Dump release

Our quarterly publication of the Stack Exchange Network data (the “data dumps”) is currently delayed. While the process of copying and uploading data was scheduled to begin on Sunday, December 3rd, we’...
Rosie's user avatar
  • 14k
1 vote
0 answers
152 views

Why is the 'Favorite' count so low in the Stack Overflow data? [duplicate]

In the Stack Overflow database, there is a column called "favorite count". I have found that favorite count indicates users find the question interesting or save it for later review. I have ...
I192058 Misbah Minhas's user avatar
5 votes
0 answers
248 views

How can I get notified about released Stack Exchange dumps?

I am interested in being notified when a new data dump is published on the internet archive. I want to find some way to be automatically notified - ideally an RSS feed, but an email or something else ...
VLAZ's user avatar
  • 11.6k
71 votes
1 answer
1k views

Possible delay in Data Dump release - now targeting September 5th

As you probably know, it’s time for the quarterly publication of our data (affectionately known as the “data dumps”). The process of copying, uploading, and publishing the data over a notoriously ...
Jody Bailey's user avatar
  • 787
3 votes
1 answer
121 views

Discrepancy in Post Counts between Local MySQL Stack Overflow Database and Data Stack Exchange Query

I recently downloaded the Stack Overflow data from Stack Overflow datadump and imported it into my local MySQL database. While attempting to retrieve the total number of posts for the year 2021 using ...
I192058 Misbah Minhas's user avatar
65 votes
11 answers
4k views

The company's commitment to the data dumps, the API, and SEDE

Many words have been written around the company's commitment to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE). Much of that text can be confusing or ...
Philippe's user avatar
  • 21k
3 votes
1 answer
472 views

How can I download a small data set (1 year 2023-2022) from Stack Exchange?

I want to download a small dataset for one year. I didn't find a link that allows me to download 1 year of data or the last five years' data. I tried "https://data.stackexchange.com/stackoverflow/...
I192058 Misbah Minhas's user avatar
-9 votes
1 answer
260 views

Does Stack Exchange Inc. intend to sue firms that use language models that are partly trained on Stack Exchange?

Stack Exchange Inc. has expressed an interest in financially profiting from some companies that use Stack Exchange data to train their model. Many language models are partly trained on Stack Exchange ...
Franck Dernoncourt's user avatar
4 votes
1 answer
170 views

Why are the profileimageurl in the data dump of almost all Stack Exchange websites blank?

I wonder why the profileimageurl in the data dump of almost all Stack Exchange websites are blank.
Ambitiouscoder's user avatar
2 votes
2 answers
194 views

Does Stack Exchange offer any incentives related to use of their data explorer in research?

I'm interested in whether or not there have historically been this sort of partnership with research in the academic space or if it is planned for the future. This could be something like a grant, ...
William Ledbetter's user avatar
-11 votes
0 answers
1k views

What is the current (June 2023) status of the data dumps and the company’s commitment to them? [duplicate]

What is the current (June 2023) status of the data dumps, the API, and the Data Explorer? Will the company continue to provide them going forward?
Philippe's user avatar
  • 21k
14 votes
2 answers
983 views

Is SE in violation of the CC BY-SA 4.0 license by not providing data dumps?

SE is no longer providing data dumps for the content contributed by their users. CC BY-SA 4.0 states: No additional restrictions — You may not apply legal terms or technological measures that legally ...
CharonX's user avatar
  • 1,041
36 votes
0 answers
281 views

Is the 'restrictions'/'guardrails' on dump access going to affect the Overflow Offline project?

I've generally been interested in the CSR aspect of Stack Overflow Inc. One of the initiatives that Stack Overflow Inc. was working on was the Overflow Offline project. As per your own media - it ...
Journeyman Geek's user avatar
40 votes
0 answers
225 views

Stopping data dumps to "protect data" - harmful and threatening to the community

I find it a bit odd that "senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs", but historically has only cared about ...
Thomas Owens's user avatar
  • 50.5k
11 votes
0 answers
236 views

What initiatives exist to create a publicly available dump of SE under the CC BY-SA 4.0 license?

SE is not providing a dump for the time being. They may do it again in the future, but with some restrictive ToS/license/etc. This makes me wonder: What initiatives exist to create a publicly ...
Franck Dernoncourt's user avatar
44 votes
2 answers
878 views

How can SE gate access to the Dump that will allow individuals access to the data while preventing "misuse" by for-profit organizations?

I read this answer written by Jody Bailey: We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations ...
Franck Dernoncourt's user avatar
3 votes
0 answers
208 views

When does the SE "senior leadership" plan to re-enable the uploads of the data dump to Archive.org? [duplicate]

I read this answer written by AMtwo: The job that uploads the data dump to Archive.org was disabled on 28 March, and marked to not be re-enabled without approval of senior leadership. This is a huge ...
Franck Dernoncourt's user avatar
26 votes
2 answers
468 views

Could someone legally replace Stack Exchange in posting the data dumps to the Internet Archive?

Recently the user discovered that without any official communication the company stopped posting the monthly data dump to the Internet Archive web site. Causes are currently unknown but some users ...
SPArcheon's user avatar
  • 29.4k
97 votes
0 answers
689 views

Will SE shut down the free API for good? [duplicate]

SE has intentionally disabled the data dump. And the CEO has stated to the press that he plans to charge AI companies for access to the data. Given these developments I think it is almost inevitable ...
Mad Scientist's user avatar
310 votes
14 answers
39k views

June 2023 Data Dump is missing

The data dump usually gets refreshed the first weekend of the month, every 3 months. The current data dump is still from March. Is there just a problem and it's delayed like in the past? Relevant ...
Data Dude's user avatar
  • 1,801
1 vote
0 answers
31 views

Where is the GitHub link in the Data Exchange Explorer located? [duplicate]

I went closely over the SEDE's fields, but couldn't find where the GitHub link is saved for a user: I ran this query on my own profile but couldn't find my GitHub account.
abbood's user avatar
  • 383
0 votes
0 answers
60 views

Why isn't the summation of a user's score equal to their reputation? [duplicate]

I was going over the database schema documentation and was trying to understand how the Post scoring system works. I was expecting this query: select sum(p.score) from posts p where p.OwnerUserId = ...
abbood's user avatar
  • 383
4 votes
1 answer
64 views

What is the difference between DisplayName and Id on the Stack Exchange Data Explorer Users table?

I am trying to pull a list of all users who have commented on an answer and tie it back to their User profile, if one exists. However, not all Comments table records have a UserId associated with them ...
William Ledbetter's user avatar
12 votes
1 answer
197 views

Update the readme.txt of the data dump ... again

In 2017 I asked to update the readme.txt of the data dump. Today I revisited that read me due to Stack Exchange Data Dump — OwnerDisplayName and LastEditorDisplayName. I was surprised to find that the ...
rene's user avatar
  • 89.5k
5 votes
1 answer
113 views

Stack Exchange Data Dump — OwnerDisplayName and LastEditorDisplayName

I'm using the Stack Exchange Data Dump and am trying to have posts along with the creators of the posts. I would like to put text like "created on [date] by [user]". In most cases, the ...
busterroni's user avatar
12 votes
2 answers
383 views

Issues decompressing the October/2022 dump

I'm having issues decompressing the October/2022 stackoverflow.com-Posts.7z published at https://archive.org/download/stackexchange. The command works for older dumps, but it fails for this. Is there ...
andreban's user avatar
  • 221
14 votes
1 answer
319 views

When will the October 2022 data dump be ready?

For those curious about the progress of the quarterly data dump (and the automation behind it), has the current process started? Are we following the schedule described here? Release schedule for the ...
Aaron Bertrand's user avatar
  • 42.7k
6 votes
1 answer
175 views

Why are there values of 1 and 3 for LinkTypeId and no value of 2? (Stack Exchange data dump)

I have found an explanation for LinkTypeId 1 and 3 here. Why was the second type skipped?
Ivan Pavlov's user avatar
4 votes
1 answer
176 views

Query for retrieving all answer ids for questions which a user already answered

I'm looking for a query from a Stack Overflow dump to find this. Example: user-id 'XXXXX' answered_list_id:[10,20,30,40] Also, I don't consider whether the answer is accepted or not. FROM Users AS u ...
mj lajy's user avatar
  • 41
3 votes
1 answer
117 views

What happens to the content of a SE site that has been closed?

The SE questions/answers are under the CC BY-SA license (ref). When a beta site does not make it and is closed, what happens to its content? Is it lost forever or is there a way to access the CC BY-SA ...
Noil's user avatar
  • 386
8 votes
1 answer
158 views

Stack Overflow data dump: Discrepancy of PostIDs in Posts.xml and Comments.xml

I am aggregating the URLs that have been shared on SO, and linking them back to the corresponding tags in the Question post. Along the way, I noticed a subset of ~100 PostIds that have URLs in the ...
Lawrence Berardelli's user avatar
6 votes
1 answer
175 views

Is the post body of the Stack Exchange data dump sanitized?

I am building a project based on the Stack Overflow data dump. As per Database schema documentation for the public data dump and SEDE the Body of Posts / PostsWithDeleted is given as rendered HTML and ...
dCoder's user avatar
  • 163
0 votes
1 answer
294 views

Should I download stackoverflow.com-PostHistory.7z to be able to use the StackedOff app?

I'm currently trying to use that app. StackedOff - an offline StackExchange indexer and search engine During an internet GB restriction on my side, the stackoverflow.com-PostHistory.7z file is 30 GB; ...
NickRar's user avatar
  • 129
3 votes
0 answers
74 views

Were Stack Exchange Data dumps removed from Archive.org? [duplicate]

It seems that Stack Exchange data dumps are no longer available at Archive.org: This item is no longer available. Items may be taken down for various reasons, including by decision of the uploader or ...
Jukka Huhtamäki's user avatar
11 votes
0 answers
231 views

Where has the Stack Exchange official dump gone? archive.org/details/stackexchange [duplicate]

Why do I see this message after I only download StackOverflow-Posts.7z? I have tried to use a VPN, but I receive the same error message: This item is no longer available. Items may be taken down for ...
NickRar's user avatar
  • 129
18 votes
1 answer
1k views

Waiting for March 2022 dump

I'm used to finding the new dumps during the first week of a quarter, but now it's the 15th and I'm constantly refreshing https://archive.org/download/stackexchange. (hopefully everyone is doing well) ...
Felipe Hoffa's user avatar
4 votes
0 answers
102 views

Error while downloading all Stack Exchange data dumps

I’m working on the analysis and prediction of deletion questions in the Stack Overflow project. By comparing different data sets, I can retrieve all the deleted questions by using data dumps obtained ...
Muddunuri Mahesh's user avatar
4 votes
1 answer
136 views

How can a User's CreationDate be after they have posted?

I am currently analysing an Archive-Datadump of the apple.stackexchange.org. When looking on the posts.xml together with the users.xml, there is a strange anomaly. The earliest CreationDate among ...
pzfn's user avatar
  • 51
2 votes
0 answers
88 views

What is posthistory? [duplicate]

I am trying to use Stack Overflow offline and I see that there are 2 files names Posts and PostHistory in archive.org available for downloads. Can someone tell me what PostHistory is? Is it all the ...
user avatar
5 votes
2 answers
107 views

Use cron to get SE Data Explorer data when it's updated

According to this answer from 2012, the SE Data Explorer data is updated every Sunday at 3am. If true, is it possible to schedule a cron job to run every Monday and execute this query? When the query ...
WinEunuuchs2Unix's user avatar
3 votes
1 answer
56 views

Extract data for several user ids in SEDE

How do I extract data from data.stackexchange.com for several user ids? I'm trying to do select * from posts where (owneruserid = 123 or owneruserid = 456); but it only outputs the results for the ...
arthur.johnson221's user avatar
1 vote
1 answer
80 views

How to get edits of a particular question from the Stack Exchange data dump

I want to see all the edits that were made to a Stack Exchange question. In other words, how a question has changed over time, maybe by editing tags, removing grammatical errors, or something else. ...
Kadam-Tushar's user avatar
9 votes
2 answers
141 views

Is null assignment of `ClosedDate` column of Posts table from Stack Overflow Data dump correct?

In the Stack Overflow data dump, I can see some posts whose ClosedDate column from the Posts table is null, but their PostHistoryTypeId = 10 (which represents post was closed) from PostHistory tables. ...
Kadam-Tushar's user avatar
22 votes
1 answer
1k views

Data Dump in the Internet Archive is no longer available. What has happened?

The context I'm getting the following (see screenshot below) when visiting the Stack Exchange Data Dump in the Internet Archive. Here's the URL. This item is no longer available. Items may be taken ...
rdrg109's user avatar
  • 531
17 votes
0 answers
174 views

Inclusion of the tables available in SEDE in the data dump at the Internet Archive

Table of contents Prologue The feature request TL; DR Current content of the data dump Missing tables in the data dump Motivation for doing this Epilogue Prologue First of all, I would like to ...
rdrg109's user avatar
  • 531
4 votes
1 answer
243 views

How to get the data dump of all the tables (29 tables) for each site?

Prologue We all know that we can use the Stack Exchange Data Explorer to perform queries in (most if not all) of the Stack Exchange tables. The tables listed, as for [2021-08-05 Thu 18:21], are 29. ...
rdrg109's user avatar
  • 531

1
2 3 4 5
10