Related: Can we change the compression algorithm for the data dumps (although that was claimed to be done, and the compression algorithm at the time was LZMA(2?))
I noticed when looking at the data dump that:
Each site is formatted as a separate archive consisting of XML files zipped via 7-zip using bzip2 compression.
Now, let's look at some data. This is from the April 2014 version of the data dump.
For the Ask Ubuntu archive:
+---------------------+--------------+-------------------+------------+
| Compression Method | Size (in KB) | % of Uncompressed | % of bzip2 |
+---------------------+--------------+-------------------+------------+
| Uncompressed | 1,452,831 | 100% | - |
| Bzip2 (from dump) | 264,871 | 18.23% | 100% |
| LZMA2 (Ultra) | 205,852 | 14.17% | 77.72% |
| PPMd (192 MB dict) | 197,917 | 13.62% | 74.72% |
| PPMd (1024 MB dict) | 179,638 | 12.36% | 67.82% |
+---------------------+--------------+-------------------+------------+
And here's for the Stack Overflow comments archive:
+---------------------+--------------+-------------------+------------+
| Compression Method | Size (in KB) | % of Uncompressed | % of bzip2 |
+---------------------+--------------+-------------------+------------+
| Uncompressed | 7,480,475 | 100% | - |
| Bzip2 (from dump) | 1,653,054 | 22.10% | 100% |
| LZMA2 (Ultra) | 1,557,689 | 20.82% | 94.23% |
| PPMd (192 MB dict) | 1,367,473 | 18.28% | 82.72% |
| PPMd (1024 MB dict) | 1,303,004 | 17.42% | 78.82% |
+---------------------+--------------+-------------------+------------+
I'm not sure about the difference in speeds between the various algorithms, but we can see there's generally a 5-20% decrease in size when using LZMA2, and generally a 20-30% decrease in size when using PPMd. Now, the size of the data dump I have is ~20 GB. This would mean a 1 GB - 4 GB decrease in size when using LZMA2, and a 4 GB - 6 GB decrease in size when using PPMd.
I have around a 1.5 MB/s connection. This would translate to anywhere from 11 minutes to 68 minutes saved when downloading that data dump. As the data dump has likely gotten larger, this would only increase the time savings.
So, considering all of this, can we please switch the compression method used for the data dump to LZMA2 or PPMd?
