This question has already been asked in one form or another but the problem still exists at least for me.
I have been working with some of the largest dump files recently e.g.
stackoverflow.com-Posts.7z which is 9,598,631 KB compressed.
I need to parse the data and due to the lack of proper LZMA stream based API in .NET, I am forced to first extract this to get a ~48 GB XML file which I then stream over to get the stuff I want. This results in unnecessary time and storage wasted when dealing with larger dumps.
I therefore propose using BZIP2 which results in slightly less size of 9,598,624 KB but on the more productive side, I can then use a library such as DotNetZip to decompress the file as a stream resulting in not having to extract the file first before getting my hands on the data.
Just for reference, PPMD consistently gives better compression (for textual data) at the cost of much slowness but then again it has the same problem of LZMA for not having a stream based API in .NET at least not a free/OSS one.