Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

Just as an example, when the last data dump was released I created a small C# program to convert it into Protocol Buffers format, but omitting the text. The text itself is rarely interesting for data mining purposes, but obviously takes up a lot of the space.

That raises three alternative formats immediately:

  • Protocol Buffer (with .proto files obviously) including text
  • Protocol Buffer excluding text
  • XML excluding text

My guess is that not very many people are interested in the PB format, but if I were to also provide some appropriate tools to make it LINQ friendly, it may be fairly easy to create a LINQPad plugin or whatever. I find it easier to write LINQ than SQL :)

It might also be good to have a "diff" format so that people who have the previous data dumps can easily add the new ones. That would need to include deletions as well, of course - it wouldn't be a trivial thing to do (well, not for me anyway) but would make it a lot lighter each month.

I'm not suggesting that Jeff should be doing this by the way - although the no-text XML version would probably be welcome and relatively easy...

share|improve this question
JSON :P – bananakata Jul 7 '09 at 9:56
yay proto ;-p . – Marc Gravell Jul 7 '09 at 10:53
@Jon looks like is_cc_wiki missed the train, so i would expect more schema changes on post next month ... – waffles Jul 7 '09 at 23:40
CC Wiki will be in the next dump – Geoff Dalgas Jul 31 '09 at 19:29

Regarding the diff format, its going to be really hard as there is at least one new fields in the upcoming dump (is community wiki)

Personally, I am happy with the current one XML file fits all solution, it does not take too long to download over bittorrent and I am able to load it all into my local SQL server DB in about 6 minutes.

Also, I find the text quite useful, for example I used the text in the comments here.

I fear that introducing any complexity will reduce the teams inclination to work on this.

My biggest wish would be for a daily diff as opposed to a monthly diff. Cause that would allow for much faster feedback loops.

share|improve this answer
New fields would certainly merit new full dumps - but I don't expect the format to change that often. A daily diff would be interesting... – Jon Skeet Jul 7 '09 at 10:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .