2

Is anyone facing issues with the data dump (web download) from archive.org? I've tried downloading the files multiple times over the last few days and I keep running into problems with the xml. (malformed, unmatched quotes, strange control characters)

Is there a place where I can get the checksums for these files, so I can verify that the files are valid at my end before going down a rabbit hole and trying to fix things at my end when in fact the files are corrupted in transit?

  • shasum for the users file - b9a016f96cc1e246d40da0263add78f584f310b6 stackoverflow.com-Users.7z, if someone can share it for the Posts file, would be awesome. – Gaurav Nov 6 '17 at 13:22
2

The torrent file accompanying a release include hashes/checksums for every file in the data dump. For example, if you inspect the last torrent release here by opening the dev console and entering:

JSON.stringify(torrent.info.files.slice(0, 10), null, 2)

You can see CRC, MD5, and SHA-1 digests for each file:

[
  {
    "crc32": "94146a77",
    "length": 2954386,
    "md5": "b6755d311dd8446bffcbe856824265cb",
    "mtime": "1497361755",
    "path": [
      "3dprinting.stackexchange.com.7z"
    ],
    "sha1": "e7b10c592f1997dae2288a1f026efa57ff0bd4f1"
  },
  {
    "attr": "p",
    "length": 1239918,
    "path": [
      ".____padding_file",
      "1586"
    ]
  },
  {
    "crc32": "ff8cbe9b",
    "length": 335208,
    "md5": "1f03da7b68ab438a9d880b3d71bc1d73",
    "mtime": "1497369239",
    "path": [
      "Sites.xml"
    ],
    "sha1": "4ef1dda6dc84b005d99247ccc3a4ea01488cf507"
  },
  {
    "attr": "p",
    "length": 1761944,
    "path": [
      ".____padding_file",
      "1916"
    ]
  },
  {
    "crc32": "c04284cb",
    "length": 74592457,
    "md5": "a6231effeff7dd30434ff79b501a41f4",
    "mtime": "1497361802",
    "path": [
      "academia.stackexchange.com.7z"
    ],
    "sha1": "d1c0883b2804e46d568b4c3c283c00b2f68a4db0"
  },
  {
    "attr": "p",
    "length": 905015,
    "path": [
      ".____padding_file",
      "1589"
    ]
  },
  {
    "crc32": "638d98d7",
    "length": 2493464,
    "md5": "6ba01e494819cdf24ebb1a5ddc403a3c",
    "mtime": "1497361807",
    "path": [
      "ai.stackexchange.com.7z"
    ],
    "sha1": "7358f24896e338cbadd0a407a4246086545d37ae"
  },
  {
    "attr": "p",
    "length": 1700840,
    "path": [
      ".____padding_file",
      "1588"
    ]
  },
  {
    "crc32": "3dde7bf6",
    "length": 79159976,
    "md5": "6a5f8abb93dc999414b4bdf5b9a9a8e7",
    "mtime": "1497361855",
    "path": [
      "android.stackexchange.com.7z"
    ],
    "sha1": "850b87315f9a6830ff8963b35c9ec6e0c1c16797"
  },
  {
    "attr": "p",
    "length": 531800,
    "path": [
      ".____padding_file",
      "1595"
    ]
  }
]

This online tool only works for torrents I've added, but your client may have a similar option.

Unfortunately, torrent generation was accidentally disabled earlier this year. 😞

1
$ sha256sum stackoverflow.com-Posts.7z 
a03feea06a493d999b555339883aedece0b5a1f9e9267654aa707008169d6357  
stackoverflow.com-Posts.7z

And it seems to be "well-formed":

$ 7z l stackoverflow.com-Posts.7z 

Scanning the drive for archives:
1 file, 11652108189 bytes (11 GiB)

Listing archive: stackoverflow.com-Posts.7z

--
Path = stackoverflow.com-Posts.7z
Type = 7z
Physical Size = 11652108189
Headers Size = 122
Method = BZip2
Solid = -
Blocks = 1

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2017-08-31 16:48:29 ....A  58183244847  11652108067  Posts.xml
------------------- ----- ------------ ------------  ------------------------
2017-08-31 16:48:29        58183244847  11652108067  1 files

$ 7z e -so stackoverflow.com-Posts.7z | pv -s 58183244847 | xmlwf 
54.2GiB 0:21:59 [42.1MiB/s] [=============================================>] 100%  

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .