15

The recently announced Creative Commons Data Dump Jan ’11 is only about 1.22GB in contrast to the Creative Commons Data Dump Nov 10 with about 2.02GB.

Without comparing every file the most striking difference is the Stack Overflow dump, where the current 012011 Stack Overflow.7z has only 934MB in contrast to the former Stack Overflow 11-2010.7z with 1.83GB.

My initial thought has been, that the deletionists might have finally won ;) But introspecting the former yields two missing files in fact, namely badges.xml and posthistory.xml, which are still present in the other sites archives (haven't checked any single one though); in addition comments.xml is an empty file, so all in all the integrity of the data seems a bit questionable.

Could the dump get fixed and rerolled accordingly, especially in light of the (somewhat surprising) shift to a bi-monthly release cycle? Thanks!

Improve this question

2 Answers 2

Reset to default
13

This one was my fault - this was due to a failure in the automated process that creates the dump from OR. A separate automated process restores a copy of the databases from NY in OR nightly which happened in the middle of creating this export. The export creation process did not report this error or halt the entire dump which is what should have happened. I will recreate a new dump for SO and get this torrent uploaded today. If there are any other issues you find please report them here.

Improve this answer
1
  • 7
    Thanks much for recreating the dump and especially the explanation of what has happened (+1) - such transparency (and insights into operational procedures of a site network your size) is always highly appreciated :)
    – Steffen Opel
    Commented Jan 16, 2011 at 17:36
4

This is now fixed. The correct size is > 3 GB, so anything smaller than that, or anything that was downloaded before ~ 8 PM Pacific on Jan 16, 2011, should be re-downloaded.

Improve this answer
1
  • Looking good so far - thanks for the status update and the swift action!
    – Steffen Opel
    Commented Jan 17, 2011 at 8:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .