Just as an example, when the last data dump was released I created a small C# program to convert it into Protocol Buffers format, but omitting the text. The text itself is rarely interesting for data mining purposes, but obviously takes up a lot of the space.

That raises three alternative formats immediately:

  • Protocol Buffer (with .proto files obviously) including text
  • Protocol Buffer excluding text
  • XML excluding text

My guess is that not very many people are interested in the PB format, but if I were to also provide some appropriate tools to make it LINQ friendly, it may be fairly easy to create a LINQPad plugin or whatever. I find it easier to write LINQ than SQL :)

It might also be good to have a "diff" format so that people who have the previous data dumps can easily add the new ones. That would need to include deletions as well, of course - it wouldn't be a trivial thing to do (well, not for me anyway) but would make it a lot lighter each month.

I'm not suggesting that Jeff should be doing this by the way - although the no-text XML version would probably be welcome and relatively easy...

  • JSON :P
    – bananakata
    Jul 7, 2009 at 9:56
  • yay proto ;-p . Jul 7, 2009 at 10:53
  • @Jon looks like is_cc_wiki missed the train, so i would expect more schema changes on post next month ...
    – waffles
    Jul 7, 2009 at 23:40
  • CC Wiki will be in the next dump Jul 31, 2009 at 19:29

1 Answer 1


Regarding the diff format, its going to be really hard as there is at least one new fields in the upcoming dump (is community wiki)

Personally, I am happy with the current one XML file fits all solution, it does not take too long to download over bittorrent and I am able to load it all into my local SQL server DB in about 6 minutes.

Also, I find the text quite useful, for example I used the text in the comments here.

I fear that introducing any complexity will reduce the teams inclination to work on this.

My biggest wish would be for a daily diff as opposed to a monthly diff. Cause that would allow for much faster feedback loops.

  • New fields would certainly merit new full dumps - but I don't expect the format to change that often. A daily diff would be interesting...
    – Jon Skeet
    Jul 7, 2009 at 10:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .