Questions tagged [data-dump]
This tag is about the quarterly creative commons data dumps of all public data in the Stack Exchange network Q&A sites.
457
questions
357
votes
2
answers
94k
views
Database schema documentation for the public data dump and SEDE
Stack Exchange releases "data dumps" of all its publicly available content roughly every three months via archive.org, and also makes that information queryable over the Internet at the Stack Exchange ...
163
votes
31
answers
10k
views
Interesting queries on Data Explorer [closed]
Let's collect some interesting queries for the Stack Exchange Data Explorer here.
How many upvotes do I have for each tag? (how long before tag badges?)
SELECT TOP 20
TagName,
COUNT(*) AS ...
137
votes
41
answers
5k
views
Sizes and overlaps of communities on the sites
This might be totally useless, but I wanted to join the league of people who use the Google chart API and the data dump to create pretty pictures.
So I took a look at the relative sizes of the "...
76
votes
10
answers
9k
views
What interesting stats can I obtain from the Stack Overflow data-dump?
What interesting statistics have you discovered from analysing the Stack Overflow data-dump?
Related:
Blog post announcing the data-dump
Direct link to the .torrent (original XML files) (2009-06)
...
74
votes
5
answers
2k
views
Dump of my own Stack Exchange content
Is there a way to quickly save all my Stack Exchange content across all Stack Exchange sites, as hundreds of HTML pages? (for me probably 15 or 20 sites: SO, dsp.SE, unix.SE, etc.)?
The ideal way ...
- 1,494
63
votes
8
answers
3k
views
Introducing the Stack Exchange Data Explorer aka SEDE
This is the announcement post of the Stack Exchange Data Explorer. Its answers were meant to iron out the initial glitches 6 to 8 years ago.
If you have found a bug or need support today with any of ...
- 107k
60
votes
2
answers
25k
views
Where are the Stack Exchange data dumps?
I see a lot of talk around here regarding the Stack Overflow public data dump.
Where is it? How can I play with it?
Return to FAQ index
- 1,784
58
votes
6
answers
6k
views
Tag trends by week
Using the data dump, I've put together some tag trend graphs that show the rate of tagged questions over time. Below are some samples, there are many more on the linked page. I can update the page ...
- 25.8k
57
votes
3
answers
9k
views
All Stack Exchange data dumps
The original ClearBits link to the September 2011 data dump has broken. How can I access it and other previous data dumps? The Internet Archive only has the latest one.
- 727
54
votes
6
answers
670
views
Please add close flags, duplicate links and review data to the public data dump
Over the past year, a lot of meta data has been changed or added to the system:
Review tasks and results
Duplicate links (separate from close votes and ad-hoc editing conventions)
Close votes as a ...
- 432k
54
votes
1
answer
4k
views
Stack Overflow top 20 tags in ebook format
Further to a discussion over a year ago about Stack Overflow portable edition, I've since got a Kindle and have been working on this project again.
I got a fast new machine with lots of disk space, ...
- 25.8k
52
votes
10
answers
14k
views
Scripts to convert data-dump to other formats
Inspired by this question, perhaps we could collect together a bunch of scripts to convert the XML data-dump files into other formats, such as SQLite, MySQL and the likes.
To keep things consistent, ...
41
votes
8
answers
4k
views
Stack Overflow stats graphs
Once the Stack Overflow data dump became available, I put together a data analysis script and created some graphs of SO activity and history. I thought this new meta site would be an appropriate place ...
- 25.8k
39
votes
1
answer
20k
views
Is there a direct download link with a raw data dump of Stack Overflow? (not a torrent)
I would like to create a local MySQL copy of Stack Overflow for academic purposes (school project). As far as I know, the database dump of the
stackoverflow.com database is available only in an XML ...
- 539
39
votes
2
answers
3k
views
Interactive tag trend comparison page
My Sunday afternoon project this weekend was to add a bit of jQuery pixie dust to the tag trend graph page. The new Stack Overflow Tag Trends page allows you to choose which tags to compare and ...
- 25.8k
39
votes
0
answers
293
views
Why does the SEDE ReviewTaskResults table not include user data, which is all public anyway?
I wanted to do some data mining on the SEDE, and I ran across a limitation which I cannot understand.
The Data Explorer table ReviewTaskResults currently contains the following information:
Id
...
- 18.4k
35
votes
1
answer
1k
views
Can the Stack Exchange data dump be provided as a public data set on Amazon Web Services (AWS)?
Amazon provides Public Data Sets on Amazon Web Services (AWS):
Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based ...
- 25.8k
32
votes
18
answers
1k
views
Don't forget to mention.... SQL injection!
I've thought for a long time that answers that mention SQL injection get upvoted a lot, even if SQL injection is entirely unrelated to the question. It looks like the new data browser supports that ...
- 6,153
32
votes
3
answers
2k
views
Stack Overflow portable edition
Stack Overflow is quickly reaching a point where any question that you might want to ask, may already have been asked. Wouldn't it be nice to be able to take this resource with you, even in an offline ...
- 25.8k
31
votes
2
answers
7k
views
Database Diagram of Stack Exchange model?
I went over the Archive site where I could retrieve data from Astronomy.SE. After I downloaded the content, I found 8 XML files that came maybe from a dump process.
Now I am not able to assemble all ...
- 445
31
votes
1
answer
684
views
Easy way to make local copy of entire SE database?
Recently, I have been using the Data Explorer to try to find fishy users on SO. However, the Data Explorer is limited when it comes to cross-site and time-consuming queries, so I want to create a ...
- 1,159
30
votes
2
answers
3k
views
Meaning of Values for PostTypeId in data explorer or in data-dump [duplicate]
I am looking for some explanation of the schema for the data dump of se sites.
Specifically I'd like to know the values for PostTypeId field (1=Questions?, 2=Answers?, 3=?, etc).
I have tried to ...
- 697
30
votes
6
answers
1k
views
StackQL: Yet Another Data Dump Sandbox [closed]
StackQL iswas a direct-query site for the public data dump.
Like the original, JavaScript is required, and it's limited to 1000 rows in the result set. (You can bust out of that if you know how, but ...
26
votes
1
answer
808
views
Stack Exchange Data Dump no longer available via BitTorrent (since June 2017)
The Data Dump on Archive.org was last updated a month ago, but the torrent version is not available. In fact, no torrent has been made available for any release since June (see my index), when the ...
- 1
26
votes
2
answers
489
views
Stack Exchange and Stack Overflow in eBook Format
I just finished a project to convert SE data-dumps into draft PDF eBooks, which I'm calling SE2go. SE2go eBooks are interactive, they contain all the original links within the question and answer body,...
- 417
25
votes
4
answers
779
views
Options for community data dump
We are in the process of automating the community data dump process and have reached the stage where we need to decide how and where to host the exported files.
We expect to have one data dump for all ...
- 153k
25
votes
1
answer
598
views
When will quarterly data dumps resume?
According to this blog article dated Sept 2011: https://blog.stackoverflow.com/2011/09/creative-commons-data-dump-sep-11/
While we will always continue to produce Stack Exchange creative commons ...
- 405
25
votes
1
answer
200
views
Add UserId information for bounty starters and closers
Currently the "UserId" info is stripped from the Votes table for "BountyStart" and "BountyClose" vote types (i.e. for VoteTypeId=8 and VoteTypeId=9 respectively, check an ...
- 3,142
25
votes
0
answers
159
views
Please add the individual close-as-off-topic reason in the Data Explorer
A couple of previous questions (How to find the actual close reason for a question? and How to determine why a question was closed from the data dump/data explorer) document the fact that it is ...
- 18.4k
23
votes
5
answers
750
views
Benfords law as it relates to SO reputation [closed]
This isn't really a question but I thought that it may interest people here to see that SO reputation seems to follow Benford's law
Here's the data I got from statoverflow (I omitted accounts with ...
- 2,424
23
votes
9
answers
722
views
What makes a SOpedian's profile worth visiting? (Data dump analysis)
After reading folks' posts here on MSO about the prominent personalities on SO (Atwood, Spolsky, Skeet, Gravell, Rich B, et al) and the impact on reputation, I decided to "run some number", ...
- 12k
23
votes
1
answer
4k
views
Is there a problem with the ClearBits data dump?
I tried to connect to http://www.clearbits.net/creators/146-stack-exchange-data-dump and got an error message:
ERR_ADDRESS_UNREACHABLE.
Is this a known problem? Do you expect that the problem ...
- 241
22
votes
1
answer
1k
views
Crawling Stack Overflow instead of the dumps for fresher data in our search results
We have a search utility on Yfrog that indexes all tweets that hit yfrog.com (we have 1.5 billion tweets indexed). We want to add all Stack Overflow questions to our index so that we could refer ...
- 231
22
votes
1
answer
760
views
Data Dump in the Internet Archive is no longer available. What has happened?
The context
I'm getting the following (see screenshot below) when visiting the Stack Exchange Data Dump in the Internet Archive. Here's the URL.
This item is no longer available.
Items may be taken ...
- 511
22
votes
0
answers
658
views
Regular Stack Exchange Data Dumps
The Stack Exchange Data Explorer (SEDE) is now updated on a weekly basis. It would be nice if all of that public data was made accessible off-line on a regular and consistent basis. The SEDE says that ...
- 34.3k
21
votes
3
answers
205
views
Linkback mechanism for questions?
I see every once in a while a phenomenon that makes me wish that SO had a linkback mechanism, concretely to make a triple (SO_URL, HTTP_Referrer, Timestamp) part of the public data dump.
The reason ...
- 10.4k
21
votes
0
answers
370
views
Adding chat transcripts as an available data dump
Since the chat system has received a bit more love recently (thank you @balpha), it would appear now is the time to make any suggestions regarding additional functionality.
Chat is still an important ...
- 3,136
20
votes
1
answer
752
views
Hosting all Stack Exchange data dumps on archive.org
Currently Stack Exchange only hosts the most recent dump on archive.org (Where are the Stack Exchange data dumps?).
The previous dumps are only available through torrents: All Stack Exchange data ...
- 23.3k
20
votes
1
answer
984
views
Can we change the compression algorithm for the data dumps
Last time the data dump went out (April), I noticed it was quite large. And I can only guess that the next one will be much bigger. I noticed that the default LZMA compression algorithm is being used....
- 1,823
19
votes
2
answers
620
views
Why is vote time missing in the SE data dump and SEDE?
Is there a particular reason why vote times are omitted in the data dump?
IMO this could be a very useful data-set. Is this one of the steps that was done to anonymize the data, or does it simply not ...
- 14.5k
19
votes
3
answers
770
views
Data dump, Programmers and Software Engineering
I've removed the poll style answers in favour of open discussion.
I realise now this isn't the best way to have a proper discussion about this issue - sorry to anyone who commented/voted.
This is ...
- 153k
19
votes
1
answer
184
views
Please include Beta Stack Exchange sites in the data dump
In this blog post it is stated that:
The latest version of the Stack Exchange Creative Commons Data Dump is
now available. This reflects all public data in …
Stack Overflow
Server Fault
Super User
...
- 4,652
19
votes
3
answers
241
views
The mystery of the tagged answer - and more
It is (currently) not possible to tag answers. Yet, if I query Data Explorer, then one answer does have a tag, namely javascript. I'm at a loss: how did that happen?†
SELECT Id, Tag FROM Posts,...
- 5,359
17
votes
1
answer
799
views
Does archival interfere with attribution removal?
There was a question asking about CC BY-SA 3.0's requirement to remove attribute on request. It was brought up that automatic archives may still preserve historical information, including attribution ...
17
votes
3
answers
20k
views
Where I can find Stack Overflow’s open source dataset
Based on the info posted here,
I realize that there is a
Stack Overflow’s open source dataset
How can I get it?
- 293
17
votes
3
answers
255
views
Include all sites in future data dumps
This issue was brought up, with regard to ServerFault, nearly three months ago. At that time, Jeff responded:
We will do this, but I agree with Michael -- there's not enough data yet. Plus, we'd like ...
- 37.5k
17
votes
2
answers
144
views
Should we include information on whether post activity was from one of the mobile apps in the public data?
Internally, we are discussing making it easier for us to tell if post activities came via the mobile version of the site. We already have a column in the PostHistory table called ApplicationId that ...
- 74.8k
17
votes
2
answers
598
views
Stack Overflow Outdated Dump
I downloaded Stack Overflow posts dump ('stackoverflow.com-Posts.7z') on 13 Sept 2019 which was lastly modified on 4 Sept 2019 according to https://archive.org. However, after I extracted the data I ...
- 173
17
votes
0
answers
115
views
Stack Exchange Data Explorer and Stack Exchange Data Dumps should include public chat room messages
All data from public chat rooms should be available in the data dumps and queryable in the Stack Exchange Data Explorer. Of course, deleted messages should not be available (since the general public ...
- 34.3k
17
votes
0
answers
561
views
Stack Exchange September 2014 Data Dump is available
The complete Stack Exchange data dump up to September 2014 is ready and can be found hosted on the Internet Archive. The latest can always be found here:
https://archive.org/details/stackexchange
...
- 48.4k