7

From what I understand, the public database dump "should" contain any information you could gather by scraping the site. As far as I can tell, revision histories are not included. Is there a good reason for this omission?

Edit: I actually asked this question mostly because I'm preparing a public dump of a StackExchange site and modeling it closely after the SO dump. I feel like people should have access to anything they could scrape from the site, but I wanted to know if there are likely to be any serious problems.

If the non-public S[OFU] data is similar to my data, including the revision history would increase the size of the dump by far less than a factor of two and wouldn't complicate things very much. On my end, it looks like there is a separate table for revision history and current posts. So one easy solution is to just include the revision history table. Another possibility is to just include the revision history table, which would only slightly increase the size of the dump, but it means that if you want to query a post, you would have to search for the last row with a given PostId.

Improve this question

2 Answers 2

Reset to default
4

Revision details are now included with the new PostHistory XML which will be out soon. Please see Anatomy of a Data Dump for a description of all the new fields.

Improve this answer
2

I am sympathetic to this idea, but it will add considerable complexity to either the table structure of the dumps or the queries needed to do reliable data mining on the dump.

Do we have any idea how often people are frustrated in their data mining goals by not having the revision info (i.e. speak up if you've been stymied in the past)?

Improve this answer
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .