Data dump, Programmers and Software Engineering

Question

I've removed the poll style answers in favour of open discussion.

I realise now this isn't the best way to have a proper discussion about this issue - sorry to anyone who commented/voted.

---

This is mostly a followup on this question:

Why no softwareengineering.stackexchange data in Stack Exchange Data Dump (March 14, 2017)

As I've answered, the reason this is happening is that the root database (in the production Stack Exchange database server) is still called StackExchange.Programmers - a database name that propagates to SEDE and through it to the data dump.

The question here is what to do about it - the current situation is somewhat broken for users of the data dump.

These are the options we have:

• Do nothing
• Update the root database name (requires downtime - so very undesirable)
• Update the update scripts so the SEDE database name is the new name, change will flow to the data dump export (breaks any queries using the old name)
• Update the export to the data dump so only it gets the new name
• Something else we haven't thought about

I have added all of these options as answers below - please vote on the one you most prefer, we will try to accommodate the top voted answer.

Answer 1

Jeremy already mentioned Beer SE in a comment, but I'm pretty sure we have a few other renames in the past as well. Moderators -> Community Building comes to mind, possibly others. Heck, Mechanics' DB is still StackExchange.Garage. We'd need to do figure out what all needs to be updated.

Given that this isn't likely gonna be the last time we rename a site in a way that affects the subdomain/database names, I think I'd prefer building some sort of aliasing thing into the dump process, so that we don't have to take a site offline every time we need to do a rename.

Or, even simpler, provide release notes with a data dump and/or note these few cases in our description on archive.org.

Answer 2

We kinda solved this problem once already... Site Names in stackexchange_files.xml

The goal here was to put the familiar (or at least, current) name of the site in Sites.xml with enough information to map that to the data. So the problem here is either:

1. Folks don't know how to use this file to establish the right mappings, or
2. The file contains inaccurate / incomplete data and thus cannot be used for the right mappings

...let's figure out which of these is a problem. Here's the current entry for Software Engineering:

<row Id="22" 
     TinyName="programmers" 
     Name="Software Engineering" 
     LongName="Software Engineering" 
     Url="http://softwareengineering.stackexchange.com/" 
     ImageUrl="https://cdn.sstatic.net/Sites/softwareengineering/img/logo.png" 
     IconUrl="https://cdn.sstatic.net/Sites/softwareengineering/img/icon-16.png" 
     DatabaseName="StackExchange.Programmers" 
     Tagline="Q&A for expert programmers interested in professional discussions on software development" 
     TagCss=".post-tag{
 background-color:#E0EAF1;
 border-bottom:1px solid #3E6D8E;
 border-right:1px solid #7F9FB6;
 color:#3E6D8E;
 font-size:90%;
 line-height:2.4;
 margin:2px 2px 2px 0;
 padding:3px 4px;
 text-decoration:none;
 white-space:nowrap;
 }
 .post-tag:hover {
 background-color:#3E6D8E;
 border-bottom:1px solid #37607D;
 border-right:1px solid #37607D;
 color:#E0EAF1;
 text-decoration:none;}" 
      TotalQuestions="44062" 
      TotalAnswers="136557" 
      TotalUsers="216079" 
      TotalComments="391424" 
      TotalTags="1626" 
      LastPost="2017-03-13T21:40:40.657" 
      BadgeIconUrl="https://cdn.sstatic.net/Sites/softwareengineering/img/apple-touch-icon.png"/>    

Now, that's enough information to render an icon, the proper name, and either resolve or compose canonical URLs for posts on that site. Also enough information to render a stylized tag, on the off-chance that's ever important.

But it lacks sufficient information to identify the file in which the data is stored!

Each site lives in a 7z archive with the name of the site's domain. Some sites - like Software Engineering - use the old domain (programmers.stackexchange.com.7z) which isn't listed in Sites.xml; others - like Motor Vehicle Maintenance and Repair - use the current domain (which can be obtained from the Url field).

Using the database name doesn't work here either; there's no garage.stackexchange.com.7z.

(there are some other head-scratching inconsistencies in this file, like the use of Statistical Analysis as the name for the site that calls itself "Cross Validated" everywhere else... But that's out of scope for this discussion.)

Proposed solution

This would all be a non-issue if we were consistent about using the canonical domain for each site as its archive name. But if we can't do that, adding an "ArchiveDomain" field to Sites.xml that would reliably map to the domain used for these files would suffice.

Answer 3

Turning some of my comments into an answer:

My gut says leave it as StackExchange.Programmers forever (especially now that it's becoming clearer to me that the main problem is user confusion, in which case, documentation). Is that mapping expensive / risky for future maintenance on SE's end? One problem with changing it is this: If anybody wants to do comparative analysis on multiple archived dumps, they have to be aware of that change. On the other hand that doesn't seem like a common use case. There may be other issues like e.g. rene's question about existing queries.

If that's not possible then my second choice is to bite the bullet and accept the down time and do the rename, no matter how long it takes. Edit: Unless the sole issue here is avoiding user confusion when reading the data dumps, in which case, I doesn't seem like all the effort required to do a rename would really be worth it to avoid a few confused meta posts that could be solved with a readme and a FAQ entry...

I don't think anything in between these two options (do nothing or full, proper conversion and rename) is a good idea. I'm a big fan of all-or-nothing.

Also another thing to consider, which may actually break a lot more than the DB name change itself, is changing SEDE query link URLs from /programmers/ (which they currently are) to /softwareengineering/. I suppose you could solve that with some 301 magic on data.stackexchange.com. (Or a mass URL replacement in posts, although that wouldn't account for external links.)

Just out of curiosity: How will you handle the API site name change, if any? It appears both programmers and softwareengineering work right now, as synonyms. That seems acceptable if it's not an issue to keep it, presuming no new site in the future is created with the name "programmers", of course.

---

In an effort to help address the primary issue and as a follow-through on my comment here I've put together a list of all sites and their various site-specific "meta" data:

• SE Site Info Google Sheet

For now it's just in a spreadsheet. This table was automatically generated by combining info from Sites.xml in the data dump, the file name list from the torrent, the API, and sys.databases in SEDE, and contains:

• Various ID numbers.
• Dump filenames.
• Site URLs and aliases.
• Site API keys.
• SEDE database names.
• A couple other things.

I am thinking of how to best present this information in an organized manner e.g. a FAQ entry, given that there's a lot of it and also there is no support for tables in posts.