Just as an example, when the last data dump was released I created a small C# program to convert it into Protocol Buffers format, but omitting the text. The text itself is rarely interesting for data mining purposes, but obviously takes up a lot of the space.
That raises three alternative formats immediately:
- Protocol Buffer (with .proto files obviously) including text
- Protocol Buffer excluding text
- XML excluding text
My guess is that not very many people are interested in the PB format, but if I were to also provide some appropriate tools to make it LINQ friendly, it may be fairly easy to create a LINQPad plugin or whatever. I find it easier to write LINQ than SQL :)
It might also be good to have a "diff" format so that people who have the previous data dumps can easily add the new ones. That would need to include deletions as well, of course - it wouldn't be a trivial thing to do (well, not for me anyway) but would make it a lot lighter each month.
I'm not suggesting that Jeff should be doing this by the way - although the no-text XML version would probably be welcome and relatively easy...
