20

Last time the data dump went out (April), I noticed it was quite large. And I can only guess that the next one will be much bigger. I noticed that the default LZMA compression algorithm is being used. However, 7-zip supports a compression algorithm called ppmd. This algorithm is much better at compressing text files, which is basically what the data dumps consist of. I did a little trial, with the Stack Overflow files from the April dump. Here's the numbers I came up with. 

Uncompressed= 16,274,007,260 bytes
LZMA compressed = 3,701,540,987 bytes
PPMd compressed = 2,501,197,899 bytes

That's would give us file sizes only 67.5 % the size of using the current method being used. And since 7zip supports this format by default, all it would take to use this is to switch a single parameter when doing the compression. I'm on a limited bandwith budget with my ISP, and I'm sure others are as well, and would appreciate having smaller files.

Improve this question
6
  • What's the potential disadvantage? Will this require users to use 7-Zip to decompress the archive? It wouldn't bother me, that's one of my favorite pieces of software. But a more portable format could be worth something, if ppmd isn't one.
    – Cody Gray
    Commented May 29, 2011 at 14:30
  • It would bother me if data dumps were to require the use of a particular piece of software to read. Not that I've ever actually done so, but it's good that I have the option on any platform that I happen to want to use.
    – Donal Fellows
    Commented May 29, 2011 at 16:02
  • 3
    The files are already in 7-zip format. What else are people using to decompress it? I assume most people would be using 7-zip already. It's still compressed using 7-zip, just using a different compression algorithm. 7-zip supports many compression formats including LZMA (currently used) bzip2 and ppmd.
    – Kibbee
    Commented May 29, 2011 at 16:15
  • And p7zip should run on anything POSIXish. I see no reason this should be a problem for anybody unless they just happen to be using a relatively old 7zip version.
    – Nicholas Knight
    Commented May 30, 2011 at 0:44
  • You really should look at PAQ if you want some serious compression.
    – Engineer2021
    Commented Jun 13, 2011 at 13:05
  • 1
    @Changeling. PAQ is interesting, but Wikipedia gives some interesting information. PAQ8HP8 required 64639 seconds to compress the same data set that PPMD did in 880 seconds. They achieved 13.34% and 18.4% compression of original size respectively. So while PAQ does get considerably better compression, it's many many many times slower. See the wikipedia article for more info. (en.wikipedia.org/wiki/PAQ)
    – Kibbee
    Commented Jun 13, 2011 at 19:16

1 Answer 1

Reset to default
9

I'm not seeing this huge difference.

(compress shared files is selected in all cases, 7zip 9.20 x64 on Windows)

On the Super User dump from 11-2010:

  • uncompressed: 464 MB
  • zip: 122 MB
  • bzip: 93.7 MB
  • lzma (ultra): 79.4 MB
  • lzma2 (ultra): 79.9 MB
  • ppmd (ultra): 73.1 MB

On the Server Fault dump from 04-2011:

  • uncompressed: 638 MB
  • lzma (ultra): 110 MB
  • ppmd (ultra): 102 MB

On the Stack Overflow dump from 04-2011:

  • uncompressed: 15.1 GB
  • lzma2 (normal): 2.71 GB
  • ppmd (normal): 2.50 GB

That means PPMD consistently produces a file about 9% smaller than LZMA. Nice, but not hugely impressive..

Improve this answer
9
  • Maybe it depends on the data set. 11-2010 doesn't seem very recent. The PostHistory Table has been added since then, hasn't it? StackOverflow is by far the bulk of the data, and my test show that the difference on the StackOverflow data is startling. Can you test with the same data set, and see if you get the same. I'm using 7-zip 64 bit on windows 7.
    – Kibbee
    Commented Jun 13, 2011 at 17:10
  • @kibbee per my edit, still not seeing it. There's nothing magical about the SO data. Are you sure you had "compress shared files" checked? If you did not, this could cause the results you quoted in your question ..
    – Jeff Atwood
    Commented Jun 13, 2011 at 22:14
  • 1
    I tried the Server Fault files from the 04-2011 data dump, and I see the same numbers you do. I don't know what makes the SO data so special, but here's a screen shot of the 7-zip properties for my PPMD version. It clearly shows that all 16 GB of data are in that file, and that the final compressed size is 2.5 GB. kibbee.ca/piwigo/upload/2011/06/13/20110613183047-edd9cdf4.jpg
    – Kibbee
    Commented Jun 14, 2011 at 1:51
  • Also worth noting is that the Serverfault files "as downloaded" were 120 MB. I'm guessing that not the highest compression level was used when generating the data dumps. Perhaps this accounts for some of the differences. I'll get back later once I have time to recompress the SO data set with LZMA on Ultra setting.
    – Kibbee
    Commented Jun 14, 2011 at 2:11
  • @kibbee updated. Can we put this thing to bed now? 9% it is. (and before you worry, lzma2 is the same as lzma but with better multi-threading support so it goes faster..)
    – Jeff Atwood
    Commented Jun 14, 2011 at 7:20
  • 2
    If that 9% is free, it's still worth doing though, right? Does lzma compress significantly faster?
    – Blorgbeard
    Commented Jun 14, 2011 at 7:54
  • 1
    Ok, I really don't care if we use PPMD. Only gives 9% as long as we use the highest compression level for LZMA. The StackOverflow files in the actual data dump provided were 3.4 GB. Which is significantly different than 2.71 you quoted for LZMA above. Which is why I was pushing so much for the change. The files, as provided in the actual data dump had very low compression. I think if you want to stick with LZMA, that's fine, but I think we should be using LZMA to it's full capacities if we do.
    – Kibbee
    Commented Jun 14, 2011 at 9:28
  • Although I'm with @Blorgbeard. 9% is still 9%. But I'll leave this alone for now. I'm really not that worried about it. I just thought we could save everybody quite a bit of bandwidth.
    – Kibbee
    Commented Jun 14, 2011 at 9:37
  • 2
    @kibbee we'll switch to this for the next dump, so 2 months from now.
    – Jeff Atwood
    Commented Jun 14, 2011 at 9:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .