Questions tagged [data-dump]

This tag is about the quarterly creative commons data dumps of all public data in the Stack Exchange network Q&A sites.

Filter by
Sorted by
Tagged with
357 votes
2 answers
94k views

Database schema documentation for the public data dump and SEDE

Stack Exchange releases "data dumps" of all its publicly available content roughly every three months via archive.org, and also makes that information queryable over the Internet at the Stack Exchange ...
163 votes
31 answers
10k views

Interesting queries on Data Explorer [closed]

Let's collect some interesting queries for the Stack Exchange Data Explorer here. How many upvotes do I have for each tag? (how long before tag badges?) SELECT TOP 20 TagName, COUNT(*) AS ...
137 votes
41 answers
5k views

Sizes and overlaps of communities on the sites

This might be totally useless, but I wanted to join the league of people who use the Google chart API and the data dump to create pretty pictures. So I took a look at the relative sizes of the "...
76 votes
10 answers
9k views

What interesting stats can I obtain from the Stack Overflow data-dump?

What interesting statistics have you discovered from analysing the Stack Overflow data-dump? Related: Blog post announcing the data-dump Direct link to the .torrent (original XML files) (2009-06) ...
74 votes
5 answers
2k views

Dump of my own Stack Exchange content

Is there a way to quickly save all my Stack Exchange content across all Stack Exchange sites, as hundreds of HTML pages? (for me probably 15 or 20 sites: SO, dsp.SE, unix.SE, etc.)? The ideal way ...
user avatar
  • 1,494
63 votes
8 answers
3k views

Introducing the Stack Exchange Data Explorer aka SEDE

This is the announcement post of the Stack Exchange Data Explorer. Its answers were meant to iron out the initial glitches 6 to 8 years ago. If you have found a bug or need support today with any of ...
user avatar
  • 107k
60 votes
2 answers
25k views

Where are the Stack Exchange data dumps?

I see a lot of talk around here regarding the Stack Overflow public data dump. Where is it? How can I play with it? Return to FAQ index
user avatar
  • 1,784
58 votes
6 answers
6k views

Tag trends by week

Using the data dump, I've put together some tag trend graphs that show the rate of tagged questions over time. Below are some samples, there are many more on the linked page. I can update the page ...
user avatar
  • 25.8k
57 votes
3 answers
9k views

All Stack Exchange data dumps

The original ClearBits link to the September 2011 data dump has broken. How can I access it and other previous data dumps? The Internet Archive only has the latest one.
user avatar
54 votes
6 answers
670 views

Please add close flags, duplicate links and review data to the public data dump

Over the past year, a lot of meta data has been changed or added to the system: Review tasks and results Duplicate links (separate from close votes and ad-hoc editing conventions) Close votes as a ...
user avatar
  • 432k
54 votes
1 answer
4k views

Stack Overflow top 20 tags in ebook format

Further to a discussion over a year ago about Stack Overflow portable edition, I've since got a Kindle and have been working on this project again. I got a fast new machine with lots of disk space, ...
user avatar
  • 25.8k
52 votes
10 answers
14k views

Scripts to convert data-dump to other formats

Inspired by this question, perhaps we could collect together a bunch of scripts to convert the XML data-dump files into other formats, such as SQLite, MySQL and the likes. To keep things consistent, ...
41 votes
8 answers
4k views

Stack Overflow stats graphs

Once the Stack Overflow data dump became available, I put together a data analysis script and created some graphs of SO activity and history. I thought this new meta site would be an appropriate place ...
user avatar
  • 25.8k
39 votes
1 answer
20k views

Is there a direct download link with a raw data dump of Stack Overflow? (not a torrent)

I would like to create a local MySQL copy of Stack Overflow for academic purposes (school project). As far as I know, the database dump of the stackoverflow.com database is available only in an XML ...
user avatar
  • 539
39 votes
2 answers
3k views

Interactive tag trend comparison page

My Sunday afternoon project this weekend was to add a bit of jQuery pixie dust to the tag trend graph page. The new Stack Overflow Tag Trends page allows you to choose which tags to compare and ...
user avatar
  • 25.8k
39 votes
0 answers
293 views

Why does the SEDE ReviewTaskResults table not include user data, which is all public anyway?

I wanted to do some data mining on the SEDE, and I ran across a limitation which I cannot understand. The Data Explorer table ReviewTaskResults currently contains the following information: Id ...
user avatar
  • 18.4k
35 votes
1 answer
1k views

Can the Stack Exchange data dump be provided as a public data set on Amazon Web Services (AWS)?

Amazon provides Public Data Sets on Amazon Web Services (AWS): Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based ...
user avatar
  • 25.8k
32 votes
18 answers
1k views

Don't forget to mention.... SQL injection!

I've thought for a long time that answers that mention SQL injection get upvoted a lot, even if SQL injection is entirely unrelated to the question. It looks like the new data browser supports that ...
user avatar
  • 6,153
32 votes
3 answers
2k views

Stack Overflow portable edition

Stack Overflow is quickly reaching a point where any question that you might want to ask, may already have been asked. Wouldn't it be nice to be able to take this resource with you, even in an offline ...
user avatar
  • 25.8k
31 votes
2 answers
7k views

Database Diagram of Stack Exchange model?

I went over the Archive site where I could retrieve data from Astronomy.SE. After I downloaded the content, I found 8 XML files that came maybe from a dump process. Now I am not able to assemble all ...
user avatar
  • 445
31 votes
1 answer
684 views

Easy way to make local copy of entire SE database?

Recently, I have been using the Data Explorer to try to find fishy users on SO. However, the Data Explorer is limited when it comes to cross-site and time-consuming queries, so I want to create a ...
user avatar
  • 1,159
30 votes
2 answers
3k views

Meaning of Values for PostTypeId in data explorer or in data-dump [duplicate]

I am looking for some explanation of the schema for the data dump of se sites. Specifically I'd like to know the values for PostTypeId field (1=Questions?, 2=Answers?, 3=?, etc). I have tried to ...
user avatar
  • 697
30 votes
6 answers
1k views

StackQL: Yet Another Data Dump Sandbox [closed]

StackQL iswas a direct-query site for the public data dump. Like the original, JavaScript is required, and it's limited to 1000 rows in the result set. (You can bust out of that if you know how, but ...
26 votes
1 answer
808 views

Stack Exchange Data Dump no longer available via BitTorrent (since June 2017)

The Data Dump on Archive.org was last updated a month ago, but the torrent version is not available. In fact, no torrent has been made available for any release since June (see my index), when the ...
user avatar
  • 1
26 votes
2 answers
489 views

Stack Exchange and Stack Overflow in eBook Format

I just finished a project to convert SE data-dumps into draft PDF eBooks, which I'm calling SE2go. SE2go eBooks are interactive, they contain all the original links within the question and answer body,...
user avatar
25 votes
4 answers
779 views

Options for community data dump

We are in the process of automating the community data dump process and have reached the stage where we need to decide how and where to host the exported files. We expect to have one data dump for all ...
user avatar
  • 153k
25 votes
1 answer
598 views

When will quarterly data dumps resume?

According to this blog article dated Sept 2011: https://blog.stackoverflow.com/2011/09/creative-commons-data-dump-sep-11/ While we will always continue to produce Stack Exchange creative commons ...
user avatar
  • 405
25 votes
1 answer
200 views

Add UserId information for bounty starters and closers

Currently the "UserId" info is stripped from the Votes table for "BountyStart" and "BountyClose" vote types (i.e. for VoteTypeId=8 and VoteTypeId=9 respectively, check an ...
user avatar
  • 3,142
25 votes
0 answers
159 views

Please add the individual close-as-off-topic reason in the Data Explorer

A couple of previous questions (How to find the actual close reason for a question? and How to determine why a question was closed from the data dump/data explorer) document the fact that it is ...
user avatar
  • 18.4k
23 votes
5 answers
750 views

Benfords law as it relates to SO reputation [closed]

This isn't really a question but I thought that it may interest people here to see that SO reputation seems to follow Benford's law Here's the data I got from statoverflow (I omitted accounts with ...
user avatar
  • 2,424
23 votes
9 answers
722 views

What makes a SOpedian's profile worth visiting? (Data dump analysis)

After reading folks' posts here on MSO about the prominent personalities on SO (Atwood, Spolsky, Skeet, Gravell, Rich B, et al) and the impact on reputation, I decided to "run some number", ...
user avatar
23 votes
1 answer
4k views

Is there a problem with the ClearBits data dump?

I tried to connect to http://www.clearbits.net/creators/146-stack-exchange-data-dump and got an error message: ERR_ADDRESS_UNREACHABLE. Is this a known problem? Do you expect that the problem ...
user avatar
  • 241
22 votes
1 answer
1k views

Crawling Stack Overflow instead of the dumps for fresher data in our search results

We have a search utility on Yfrog that indexes all tweets that hit yfrog.com (we have 1.5 billion tweets indexed). We want to add all Stack Overflow questions to our index so that we could refer ...
user avatar
22 votes
1 answer
760 views

Data Dump in the Internet Archive is no longer available. What has happened?

The context I'm getting the following (see screenshot below) when visiting the Stack Exchange Data Dump in the Internet Archive. Here's the URL. This item is no longer available. Items may be taken ...
user avatar
  • 511
22 votes
0 answers
658 views

Regular Stack Exchange Data Dumps

The Stack Exchange Data Explorer (SEDE) is now updated on a weekly basis. It would be nice if all of that public data was made accessible off-line on a regular and consistent basis. The SEDE says that ...
user avatar
  • 34.3k
21 votes
3 answers
205 views

Linkback mechanism for questions?

I see every once in a while a phenomenon that makes me wish that SO had a linkback mechanism, concretely to make a triple (SO_URL, HTTP_Referrer, Timestamp) part of the public data dump. The reason ...
user avatar
  • 10.4k
21 votes
0 answers
370 views

Adding chat transcripts as an available data dump

Since the chat system has received a bit more love recently (thank you @balpha), it would appear now is the time to make any suggestions regarding additional functionality. Chat is still an important ...
user avatar
  • 3,136
20 votes
1 answer
752 views

Hosting all Stack Exchange data dumps on archive.org

Currently Stack Exchange only hosts the most recent dump on archive.org (Where are the Stack Exchange data dumps?). The previous dumps are only available through torrents: All Stack Exchange data ...
user avatar
20 votes
1 answer
984 views

Can we change the compression algorithm for the data dumps

Last time the data dump went out (April), I noticed it was quite large. And I can only guess that the next one will be much bigger. I noticed that the default LZMA compression algorithm is being used....
user avatar
  • 1,823
19 votes
2 answers
620 views

Why is vote time missing in the SE data dump and SEDE?

Is there a particular reason why vote times are omitted in the data dump? IMO this could be a very useful data-set. Is this one of the steps that was done to anonymize the data, or does it simply not ...
user avatar
  • 14.5k
19 votes
3 answers
770 views

Data dump, Programmers and Software Engineering

I've removed the poll style answers in favour of open discussion. I realise now this isn't the best way to have a proper discussion about this issue - sorry to anyone who commented/voted. This is ...
user avatar
  • 153k
19 votes
1 answer
184 views

Please include Beta Stack Exchange sites in the data dump

In this blog post it is stated that: The latest version of the Stack Exchange Creative Commons Data Dump is now available. This reflects all public data in … Stack Overflow Server Fault Super User ...
user avatar
  • 4,652
19 votes
3 answers
241 views

The mystery of the tagged answer - and more

It is (currently) not possible to tag answers. Yet, if I query Data Explorer, then one answer does have a tag, namely javascript. I'm at a loss: how did that happen?† SELECT Id, Tag FROM Posts,...
user avatar
  • 5,359
17 votes
1 answer
799 views

Does archival interfere with attribution removal?

There was a question asking about CC BY-SA 3.0's requirement to remove attribute on request. It was brought up that automatic archives may still preserve historical information, including attribution ...
user avatar
17 votes
3 answers
20k views

Where I can find Stack Overflow’s open source dataset

Based on the info posted here, I realize that there is a Stack Overflow’s open source dataset How can I get it?
user avatar
  • 293
17 votes
3 answers
255 views

Include all sites in future data dumps

This issue was brought up, with regard to ServerFault, nearly three months ago. At that time, Jeff responded: We will do this, but I agree with Michael -- there's not enough data yet. Plus, we'd like ...
user avatar
  • 37.5k
17 votes
2 answers
144 views

Should we include information on whether post activity was from one of the mobile apps in the public data?

Internally, we are discussing making it easier for us to tell if post activities came via the mobile version of the site. We already have a column in the PostHistory table called ApplicationId that ...
user avatar
  • 74.8k
17 votes
2 answers
598 views

Stack Overflow Outdated Dump

I downloaded Stack Overflow posts dump ('stackoverflow.com-Posts.7z') on 13 Sept 2019 which was lastly modified on 4 Sept 2019 according to https://archive.org. However, after I extracted the data I ...
user avatar
  • 173
17 votes
0 answers
115 views

Stack Exchange Data Explorer and Stack Exchange Data Dumps should include public chat room messages

All data from public chat rooms should be available in the data dumps and queryable in the Stack Exchange Data Explorer. Of course, deleted messages should not be available (since the general public ...
user avatar
  • 34.3k
17 votes
0 answers
561 views

Stack Exchange September 2014 Data Dump is available

The complete Stack Exchange data dump up to September 2014 is ready and can be found hosted on the Internet Archive. The latest can always be found here: https://archive.org/details/stackexchange ...
user avatar
  • 48.4k

1
2 3 4 5
10