Questions tagged [data-dump]
This tag is about the quarterly creative commons data dumps of all public data in the Stack Exchange network Q&A sites.
488
questions
4
votes
0
answers
136
views
How can I get notified about released Stack Exchange dumps?
I am interested in being notified when a new data dump is published on the internet archive. I want to find some way to be automatically notified - ideally an RSS feed, but an email or something else ...
- 8,648
68
votes
1
answer
600
views
Possible delay in Data Dump release - now targeting September 5th
As you probably know, it’s time for the quarterly publication of our data (affectionately known as the “data dumps”). The process of copying, uploading, and publishing the data over a notoriously ...
Jody BaileyStaff
- 763
3
votes
1
answer
113
views
Discrepancy in Post Counts between Local MySQL Stack Overflow Database and Data Stack Exchange Query
I recently downloaded the Stack Overflow data from Stack Overflow datadump and imported it into my local MySQL database. While attempting to retrieve the total number of posts for the year 2021 using ...
66
votes
11
answers
4k
views
The company's commitment to the data dumps, the API, and SEDE
Many words have been written around the company's commitment to the ongoing existence of the data dumps, the API, and the Stack Exchange Data Explorer (SEDE). Much of that text can be confusing or ...
- 19.7k
3
votes
1
answer
370
views
How can I download a small data set (1 year 2023-2022) from Stack Exchange?
I want to download a small dataset for one year. I didn't find a link that allows me to download 1 year of data or the last five years' data.
I tried "https://data.stackexchange.com/stackoverflow/...
-9
votes
1
answer
240
views
Does Stack Exchange Inc. intend to sue firms that use language models that are partly trained on Stack Exchange?
Stack Exchange Inc. has expressed an interest in financially profiting from some companies that use Stack Exchange data to train their model.
Many language models are partly trained on Stack Exchange ...
- 33.7k
4
votes
1
answer
161
views
Why are the profileimageurl in the data dump of almost all Stack Exchange websites blank?
I wonder why the profileimageurl in the data dump of almost all Stack Exchange websites are blank.
2
votes
2
answers
166
views
Does Stack Exchange offer any incentives related to use of their data explorer in research?
I'm interested in whether or not there have historically been this sort of partnership with research in the academic space or if it is planned for the future. This could be something like a grant, ...
-11
votes
0
answers
1k
views
What is the current (June 2023) status of the data dumps and the company’s commitment to them? [duplicate]
What is the current (June 2023) status of the data dumps, the API, and the Data Explorer? Will the company continue to provide them going forward?
- 19.7k
14
votes
2
answers
935
views
Is SE in violation of the CC BY-SA 4.0 license by not providing data dumps?
SE is no longer providing data dumps for the content contributed by their users.
CC BY-SA 4.0 states:
No additional restrictions — You may not apply legal terms or technological measures that legally ...
- 1,041
36
votes
0
answers
260
views
Is the 'restrictions'/'guardrails' on dump access going to affect the Overflow Offline project?
I've generally been interested in the CSR aspect of Stack Overflow Inc. One of the initiatives that Stack Overflow Inc. was working on was the Overflow Offline project.
As per your own media - it ...
- 165k
40
votes
0
answers
215
views
Stopping data dumps to "protect data" - harmful and threatening to the community
I find it a bit odd that "senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs", but historically has only cared about ...
- 41.4k
11
votes
0
answers
228
views
What initiatives exist to create a publicly available dump of SE under the CC BY-SA 4.0 license?
SE is not providing a dump for the time being. They may do it again in the future, but with some restrictive ToS/license/etc.
This makes me wonder: What initiatives exist to create a publicly ...
- 33.7k
43
votes
2
answers
709
views
How can SE gate access to the Dump that will allow individuals access to the data while preventing "misuse" by for-profit organizations?
I read this answer written by Jody Bailey:
We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations ...
- 33.7k
3
votes
0
answers
206
views
When does the SE "senior leadership" plan to re-enable the uploads of the data dump to Archive.org? [duplicate]
I read this answer written by AMtwo:
The job that uploads the data dump to Archive.org was disabled on 28 March, and marked to not be re-enabled without approval of senior leadership.
This is a huge ...
- 33.7k
26
votes
2
answers
446
views
Could someone legally replace Stack Exchange in posting the data dumps to the Internet Archive?
Recently the user discovered that without any official communication the company stopped posting the monthly data dump to the Internet Archive web site. Causes are currently unknown but some users ...
- 25k
98
votes
0
answers
673
views
Will SE shut down the free API for good? [duplicate]
SE has intentionally disabled the data dump. And the CEO has stated to the press that he plans to charge AI companies for access to the data.
Given these developments I think it is almost inevitable ...
- 184k
302
votes
14
answers
36k
views
June 2023 Data Dump is missing
The data dump usually gets refreshed the first weekend of the month, every 3 months.
The current data dump is still from March. Is there just a problem and it's delayed like in the past?
Relevant ...
- 1,711
1
vote
0
answers
31
views
Where is the GitHub link in the Data Exchange Explorer located? [duplicate]
I went closely over the SEDE's fields, but couldn't find where the GitHub link is saved for a user:
I ran this query on my own profile but couldn't find my GitHub account.
- 363
0
votes
0
answers
60
views
Why isn't the summation of a user's score equal to their reputation? [duplicate]
I was going over the database schema documentation and was trying to understand how the Post scoring system works. I was expecting this query:
select sum(p.score)
from posts p
where p.OwnerUserId = ...
- 363
4
votes
1
answer
58
views
What is the difference between DisplayName and Id on the Stack Exchange Data Explorer Users table?
I am trying to pull a list of all users who have commented on an answer and tie it back to their User profile, if one exists. However, not all Comments table records have a UserId associated with them ...
12
votes
1
answer
191
views
Update the readme.txt of the data dump ... again
In 2017 I asked to update the readme.txt of the data dump.
Today I revisited that read me due to Stack Exchange Data Dump — OwnerDisplayName and LastEditorDisplayName. I was surprised to find that the ...
- 85.7k
4
votes
1
answer
101
views
Stack Exchange Data Dump — OwnerDisplayName and LastEditorDisplayName
I'm using the Stack Exchange Data Dump and am trying to have posts along with the creators of the posts. I would like to put text like "created on [date] by [user]". In most cases, the ...
- 141
12
votes
2
answers
377
views
Issues decompressing the October/2022 dump
I'm having issues decompressing the October/2022 stackoverflow.com-Posts.7z published at https://archive.org/download/stackexchange.
The command works for older dumps, but it fails for this. Is there ...
- 221
14
votes
1
answer
317
views
When will the October 2022 data dump be ready?
For those curious about the progress of the quarterly data dump (and the automation behind it), has the current process started? Are we following the schedule described here?
Release schedule for the ...
Aaron BertrandStaff
- 36.6k
6
votes
1
answer
169
views
Why are there values of 1 and 3 for LinkTypeId and no value of 2? (Stack Exchange data dump)
I have found an explanation for LinkTypeId 1 and 3 here. Why was the second type skipped?
- 71
4
votes
1
answer
172
views
Query for retrieving all answer ids for questions which a user already answered
I'm looking for a query from a Stack Overflow dump to find this. Example: user-id 'XXXXX' answered_list_id:[10,20,30,40]
Also, I don't consider whether the answer is accepted or not.
FROM Users AS u
...
- 41
3
votes
1
answer
116
views
What happens to the content of a SE site that has been closed?
The SE questions/answers are under the CC BY-SA license (ref). When a beta site does not make it and is closed, what happens to its content? Is it lost forever or is there a way to access the CC BY-SA ...
- 386
8
votes
1
answer
151
views
Stack Overflow data dump: Discrepancy of PostIDs in Posts.xml and Comments.xml
I am aggregating the URLs that have been shared on SO, and linking them back to the corresponding tags in the Question post.
Along the way, I noticed a subset of ~100 PostIds that have URLs in the ...
6
votes
1
answer
166
views
Is the post body of the Stack Exchange data dump sanitized?
I am building a project based on the Stack Overflow data dump.
As per Database schema documentation for the public data dump and SEDE the Body of Posts / PostsWithDeleted is given as rendered HTML and ...
- 163
0
votes
1
answer
249
views
Should I download stackoverflow.com-PostHistory.7z to be able to use the StackedOff app?
I'm currently trying to use that app.
StackedOff - an offline StackExchange indexer and search engine
During an internet GB restriction on my side, the stackoverflow.com-PostHistory.7z file is 30 GB; ...
- 129
3
votes
0
answers
74
views
Were Stack Exchange Data dumps removed from Archive.org? [duplicate]
It seems that Stack Exchange data dumps are no longer available at Archive.org:
This item is no longer available.
Items may be taken down for various reasons, including by decision of the uploader or ...
11
votes
0
answers
231
views
Where has the Stack Exchange official dump gone? archive.org/details/stackexchange [duplicate]
Why do I see this message after I only download StackOverflow-Posts.7z?
I have tried to use a VPN, but I receive the same error message:
This item is no longer available.
Items may be taken down for ...
- 129
17
votes
1
answer
1k
views
Waiting for March 2022 dump
I'm used to finding the new dumps during the first week of a quarter, but now it's the 15th and I'm constantly refreshing https://archive.org/download/stackexchange.
(hopefully everyone is doing well)
...
- 526
4
votes
0
answers
102
views
Error while downloading all Stack Exchange data dumps
I’m working on the analysis and prediction of deletion questions in the Stack Overflow project. By comparing different data sets, I can retrieve all the deleted questions by using data dumps obtained ...
4
votes
1
answer
126
views
How can a User's CreationDate be after they have posted?
I am currently analysing an Archive-Datadump of the apple.stackexchange.org. When looking on the posts.xml together with the users.xml, there is a strange anomaly.
The earliest CreationDate among ...
- 51
2
votes
0
answers
88
views
What is posthistory? [duplicate]
I am trying to use Stack Overflow offline and I see that there are 2 files names Posts and PostHistory in archive.org available for downloads. Can someone tell me what PostHistory is?
Is it all the ...
programmer
5
votes
2
answers
104
views
Use cron to get SE Data Explorer data when it's updated
According to this answer from 2012, the SE Data Explorer data is updated every Sunday at 3am. If true, is it possible to schedule a cron job to run every Monday and execute this query?
When the query ...
- 287
3
votes
1
answer
53
views
Extract data for several user ids in SEDE
How do I extract data from data.stackexchange.com for several user ids?
I'm trying to do
select * from posts where (owneruserid = 123 or owneruserid = 456);
but it only outputs the results for the ...
1
vote
1
answer
74
views
How to get edits of a particular question from the Stack Exchange data dump
I want to see all the edits that were made to a Stack Exchange question. In other words, how a question has changed over time, maybe by editing tags, removing grammatical errors, or something else.
...
- 113
9
votes
2
answers
141
views
Is null assignment of `ClosedDate` column of Posts table from Stack Overflow Data dump correct?
In the Stack Overflow data dump, I can see some posts whose ClosedDate column from the Posts table is null, but their PostHistoryTypeId = 10 (which represents post was closed) from PostHistory tables.
...
- 113
22
votes
1
answer
1k
views
Data Dump in the Internet Archive is no longer available. What has happened?
The context
I'm getting the following (see screenshot below) when visiting the Stack Exchange Data Dump in the Internet Archive. Here's the URL.
This item is no longer available.
Items may be taken ...
- 531
17
votes
0
answers
169
views
Inclusion of the tables available in SEDE in the data dump at the Internet Archive
Table of contents
Prologue
The feature request
TL; DR
Current content of the data dump
Missing tables in the data dump
Motivation for doing this
Epilogue
Prologue
First of all, I would like to ...
- 531
4
votes
1
answer
207
views
How to get the data dump of all the tables (29 tables) for each site?
Prologue
We all know that we can use the Stack Exchange Data Explorer to perform queries in (most if not all) of the Stack Exchange tables. The tables listed, as for [2021-08-05 Thu 18:21], are 29. ...
- 531
5
votes
0
answers
126
views
Is there a data dump for SE chat logs?
I am working on a research project looking at Stackexchange posts. I am aware of the data explorer as well as the data dump stored on the internet archive. I was wondering if there was any information ...
- 51
-14
votes
1
answer
505
views
Download Stack Overflow dataset for college project [duplicate]
I would like to download the Stack Overflow dataset that contains the question title and top-rated answer (not answer id).
10
votes
1
answer
78
views
How to get "duplicates list edited" history from the public data dump or SEDE?
I want to search questions whose duplicates list has been edited. Take this question as an example:
The timeline shows that the type of "duplicates list edited" is history, but the list ...
- 101
1
vote
1
answer
98
views
ID of user in the Posts.7z public Stack Overflow dataset
I have downloaded the Posts.7z dataset from here. I have not been able to find a README file, so I go by intuition regarding the content of the file.
To my understanding, a single line of the file is ...
- 129
7
votes
1
answer
85
views
Which data dump has the information related to mentioned users in a comment
From the Stack Exchange API, I can retrieve the comments in which a specific user is mentioned using @ method.
Here is the /users/{ids}/mentioned API.
Which data dump on archive.org hosts this ...
- 231
3
votes
0
answers
119
views
Where can I find examples of Stack Exchange search queries?
Is there a dataset available to the research community that contains example queries for Stack Exchange results related to sotware? By search queries I mean "pandas to_csv tab delimiter" and ...
- 153