Timeline for Can we change the compression algorithm for the data dumps
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 14, 2011 at 9:56 | comment | added | Jeff Atwood | @kibbee we'll switch to this for the next dump, so 2 months from now. | |
| Jun 14, 2011 at 9:37 | comment | added | Kibbee | Although I'm with @Blorgbeard. 9% is still 9%. But I'll leave this alone for now. I'm really not that worried about it. I just thought we could save everybody quite a bit of bandwidth. | |
| Jun 14, 2011 at 9:28 | comment | added | Kibbee | Ok, I really don't care if we use PPMD. Only gives 9% as long as we use the highest compression level for LZMA. The StackOverflow files in the actual data dump provided were 3.4 GB. Which is significantly different than 2.71 you quoted for LZMA above. Which is why I was pushing so much for the change. The files, as provided in the actual data dump had very low compression. I think if you want to stick with LZMA, that's fine, but I think we should be using LZMA to it's full capacities if we do. | |
| Jun 14, 2011 at 7:54 | comment | added | Blorgbeard | If that 9% is free, it's still worth doing though, right? Does lzma compress significantly faster? | |
| Jun 14, 2011 at 7:20 | comment | added | Jeff Atwood | @kibbee updated. Can we put this thing to bed now? 9% it is. (and before you worry, lzma2 is the same as lzma but with better multi-threading support so it goes faster..) | |
| Jun 14, 2011 at 7:20 | history | edited | Jeff Atwood | CC BY-SA 3.0 |
added 179 characters in body
|
| Jun 14, 2011 at 2:11 | comment | added | Kibbee | Also worth noting is that the Serverfault files "as downloaded" were 120 MB. I'm guessing that not the highest compression level was used when generating the data dumps. Perhaps this accounts for some of the differences. I'll get back later once I have time to recompress the SO data set with LZMA on Ultra setting. | |
| Jun 14, 2011 at 1:51 | comment | added | Kibbee | I tried the Server Fault files from the 04-2011 data dump, and I see the same numbers you do. I don't know what makes the SO data so special, but here's a screen shot of the 7-zip properties for my PPMD version. It clearly shows that all 16 GB of data are in that file, and that the final compressed size is 2.5 GB. kibbee.ca/piwigo/upload/2011/06/13/20110613183047-edd9cdf4.jpg | |
| Jun 13, 2011 at 22:14 | comment | added | Jeff Atwood | @kibbee per my edit, still not seeing it. There's nothing magical about the SO data. Are you sure you had "compress shared files" checked? If you did not, this could cause the results you quoted in your question .. | |
| Jun 13, 2011 at 22:14 | history | edited | Jeff Atwood | CC BY-SA 3.0 |
added 94 characters in body
|
| Jun 13, 2011 at 17:10 | comment | added | Kibbee | Maybe it depends on the data set. 11-2010 doesn't seem very recent. The PostHistory Table has been added since then, hasn't it? StackOverflow is by far the bulk of the data, and my test show that the difference on the StackOverflow data is startling. Can you test with the same data set, and see if you get the same. I'm using 7-zip 64 bit on windows 7. | |
| Jun 13, 2011 at 8:04 | history | answered | Jeff Atwood | CC BY-SA 3.0 |
