7

I've downloaded the Torrent version of https://archive.org/details/stackexchange and selected only the files that I need with Transmission.

I then tried to extract with:

sudo apt-get install p7zip-full
7z x stackoverflow.com-Posts.7z

on Ubuntu 15.10 AMD64, 7z version 9.20, but it extracts up to 6.9 Gb of Posts.xml, and then gives the error:

Extracting  Posts.xml     Data Error

Sub items Errors: 1

and exit status 2.

I think this means the file is corrupted.

The other .7z I've tried extracted fine. This one is by far the largest, maybe the failure is related?

  • Ah I thought those were on topic here as I had seen others like it. Is there anywhere else I can ask? – Ciro Santilli 新疆改造中心法轮功六四事件 Nov 18 '15 at 21:16
  • Nonono they are on topic of course. Sorry, accidentally thought it was another off-topic programming question there . – nicael Nov 18 '15 at 21:16
  • Are you certain the corruption is not just local? That is, the download you have got corrupt vs the actual file in the torrent? – Oded Nov 18 '15 at 21:26
  • @Oded mmm I'll try do download again with another client. Is there a better way to check that? I've clicked verify on the Transmission client, and it said nothing. Now that I look at it, my local file is 5.7Gb, the Transmission says it should be 6.2, and the direct download on archive.org 7.3... so maybe you are right. – Ciro Santilli 新疆改造中心法轮功六四事件 Nov 18 '15 at 21:38
  • Your torrent app should be able to validate the downloaded file... – user1228 Nov 18 '15 at 21:45
  • @Won't yes, I've tried that with Transmission, it says nothing. – Ciro Santilli 新疆改造中心法轮功六四事件 Nov 18 '15 at 21:54
  • OK, I've re-downloaded and it seems to be moving forward. Gotta sleep, tomorrow I'll try to finish the decompress, sorry for the hassle. – Ciro Santilli 新疆改造中心法轮功六四事件 Nov 18 '15 at 22:35
  • I've downloaded the .7z file directly (not through bittorrent) from Archive.org twice. Both times I get an error when unpacking it. :-( – aioobe Dec 11 '15 at 6:25
3

OK, false alarm.

I downloaded the file again and it worked.

The first time I downloaded, the last 1% was very slow, maybe there is a relation?

I didn't know torrents could be so unreliable.

  • The Internet Archive, which Stack Exchange uses, is a terrible BitTorrent host for this kind of data. Their web seeds don't take into account that the torrents are updated, and will gladly send data to a client on a different version. For an appropriately-padded torrent with uncompressed files which often are unchanged between versions, this approach can be a little bit useful (though I still dislike it). But for Stack Exchanges' compressed dumps, the Internet Archive's web seeds often flood clients with tens of gigs of invalid data, interfering with the valid download and wasting bandwidth. – REINSTATE MONICA -Jeremy Banks Nov 19 '15 at 18:02
  • Unfortunately, many BitTorrent clients are more trusting of web seeds because they're provide by the torrent author, unlike the untrustworthy peer seeds they get from the network. If a peer were sending nothing but invalid data, your client would eventually block it, or at least de-prioritize it below all other seeds. But your client will continue to accept as much invalid data as a web seed will give it, assuming they're all honest transmission errors. In theory, the BitTorrent protocol should have prevented the download from being "completed" until it has the valid data, though. – REINSTATE MONICA -Jeremy Banks Nov 19 '15 at 18:20
  • I downloaded the posts and users file a few times last week and got the same error while extracting. Seems it gets corrupted a lot in transit when downloading from archive.org. I haven't found checksums for these files, but in case anyone else gets the same error, it's more likely an issue in the download than anything at your end. – Gaurav Nov 7 '17 at 19:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .