3

I am working with the StackExchange data dump, thanks ot StackExchange's splendid work, the data can not only be processed very conveniently, but also downloaded automatically, based on the files list stackexchange_files.xml. That list contains all the necessary information to download and verify (based upon total size and hashes etc.) the downloaded file, and the filenames are regular enough to figure out the StackExchange site the file applies to from the filename - at least a unique identifier for each site that matches across files where data from one site is split across multiple files.

However, one information that I would like to get is missing: The human-friendly name of each site. That name cannot be automatically extracted from the dump filenames, at least not with proper casing, as the mapping here is highly irregular (at least from a machine's point of view).

To name some examples (using the "text-only" (non-logo) title as it appears in the browser title bar):

  • stackoverflow.com -> Stack Overflow
  • codegolf.stackexchange.com -> Programming Puzzles & Code Golf Stack Exchange
  • ux.stackexchange.com -> User Experience Stack Exchange
  • bricks.stackexchange.com -> LEGO® Answers
  • cstheory.stackexchange.com -> Theoretical Computer Science Stack Exchange

Currently, the files list looks like this:

  <file name="stackoverflow.com-Users.7z" source="original">
    <format>7z</format>
    <mtime>1399387539</mtime>
    <size>92073166</size>
    <md5>be05b595ed9c79ee45e4be1fe319aaf0</md5>
    <crc32>96c113b4</crc32>
    <sha1>c0e1e4734e6f3d738497c9c0e66d940f976fb3a6</sha1>
  </file>
  <file name="bicycles.stackexchange.com.7z" source="original">
    <format>7z</format>
    <mtime>1399390862</mtime>
    <size>14820795</size>
    <md5>cf7e44ec44cb0f33351dca01160b9a2d</md5>
    <crc32>1f926140</crc32>
    <sha1>ab5b5928ee346e86117930570b3bdf19fb8372d7</sha1>
  </file>

Would it be possible to add an additional attribute or element to indicate the human-friendly site name? This would be useful for converter tools for the data dump, but also for status output during automatical downloading.

For example, like this:

  <file name="stackoverflow.com-Users.7z" source="original" label="Stack Overflow">
    <format>7z</format>
    <mtime>1399387539</mtime>
    <size>92073166</size>
    <md5>be05b595ed9c79ee45e4be1fe319aaf0</md5>
    <crc32>96c113b4</crc32>
    <sha1>c0e1e4734e6f3d738497c9c0e66d940f976fb3a6</sha1>
  </file>
  <file name="bicycles.stackexchange.com.7z" source="original" label="Bycicles Stack Exchange">
    <format>7z</format>
    <mtime>1399390862</mtime>
    <size>14820795</size>
    <md5>cf7e44ec44cb0f33351dca01160b9a2d</md5>
    <crc32>1f926140</crc32>
    <sha1>ab5b5928ee346e86117930570b3bdf19fb8372d7</sha1>
  </file>
7

Update: There is now a site metadata file we'll include in each dump called Sites.xml here: https://archive.org/details/stackexchange

It contains the following data for each site:

  • Address
  • Name
  • Description
  • ImageUrl
  • TotalQuestions
  • TotalAnswers
  • TotalUsers
  • TotalComments
  • TotalTags
  • ParentAddress (only present for child metas)

Note: These counts are not accurate for this dump because they were taken more than a week apart (SEDE updates weekly and is the source of the dump). They will accurately reflect future dumps.


Original: Unfortunately we can't do this...because we don't generate the file. This is something archive.org does on their end after the upload process finishes each time.

However, dumping something else that contains all this information may be doable - we'll take a look. It's certainly a good idea to have a master list in there.

  • Thanks - even if it's a separate XML-or-so file that contains the mapping, and even if it is incomplete, that would be helpful. FYI, I am working on a small tool to convert much of the data dump to an RDF graph for my purposes, so whenever that mapping does contain a human-friendly site-name, I could add that as an rdfs:label to the respective node representing the site in question. Should I just wait for now (and accept this answer), or are you going to follow up on this right here by extending the answer in case there is any decision? – O. R. Mapper May 15 '14 at 10:18
  • 1
    @O.R.Mapper I'm going to try and get this today - we should be able to update the existing dump with this metadata file - I'll update the answer when this is deployed. – Nick Craver May 15 '14 at 10:20
  • Incredible (in a positive way)! Fastest implementation of a (not totally trivial) feature request ever :-) Thank you! – O. R. Mapper May 16 '14 at 5:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .