2

This question has already been asked in one form or another but the problem still exists at least for me.

I have been working with some of the largest dump files recently e.g. stackoverflow.com-Posts.7z which is 9,598,631 KB compressed.

I need to parse the data and due to the lack of proper LZMA stream based API in .NET, I am forced to first extract this to get a ~48 GB XML file which I then stream over to get the stuff I want. This results in unnecessary time and storage wasted when dealing with larger dumps.

I therefore propose using BZIP2 which results in slightly less size of 9,598,624 KB but on the more productive side, I can then use a library such as DotNetZip to decompress the file as a stream resulting in not having to extract the file first before getting my hands on the data.

Just for reference, PPMD consistently gives better compression (for textual data) at the cost of much slowness but then again it has the same problem of LZMA for not having a stream based API in .NET at least not a free/OSS one.

Improve this question
8
  • Does anybody like 7zip?
    – user1228
    Oct 19, 2016 at 14:33
  • I was fairly sure it was bzip2 compressed already, just put into a .7z? See my (related, maybe duplicate) request: Use LZMA2 or PPMd instead of bzip2 for the data dump.
    – hichris123
    Oct 19, 2016 at 18:52
  • No, I can confirm it's not a BZIP2, in fact I used the 7Zip SDK to decompress the file as LZMA but it's not ideal as you would need to extract the whole file (not streamed base) and I don't want to have to do gymnastics to make LZMA work in .NET when we can hopefully get this in a better format that can make both the Stack guys and us the users happy.
    – MaYaN
    Oct 19, 2016 at 19:06
  • I... think you better check that. Since if the file only changes 7 KB by compressing it with a completely different algorithm... I find that hard to believe. Especially considering in a vast majority of instances, LZMA is much much better with compression than bzip2.
    – hichris123
    Oct 19, 2016 at 19:58
  • I don't understand what you mean?! I have tested it hence this post. As explained the proposal does not focus on any gains in the compression size but the ease of decompression as a stream so that one does not have to extract the whole 68GB of data only go through it again just to take a small part of it. The size of BZIP2 is already less (by a very small margin) than the current LZMA however BZIP2 has a much better support from the API point of view at least in .NET.
    – MaYaN
    Oct 19, 2016 at 20:05
  • What I'm trying to explain is there's a 99% chance bzip2 is already used for the compression. However, it is just stored in a 7z archive which has a different format than a normal bzip2 archive. And I can confirm that is the case, see this properties screenshot of an archive with a nice freehand circle.
    – hichris123
    Oct 19, 2016 at 23:13
  • You are absolutely right. It is indeed BZIP2 stored in a 7z format which introduces additional headers at the beginning of the file. I just cannot figure out why it would decompress using LZMA without any errors!. All I need to do now is to ignore the headers and decompress the rest. Thanks for that. Feel free to post this as an answer and I will mark as accepted.
    – MaYaN
    Oct 20, 2016 at 15:56
  • Wouldn't the xz format an alternative?
    – Braiam
    Feb 4, 2017 at 16:01

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .