17

I've removed the poll style answers in favour of open discussion.

I realise now this isn't the best way to have a proper discussion about this issue - sorry to anyone who commented/voted.


This is mostly a followup on this question:

Why no softwareengineering.stackexchange data in Stack Exchange Data Dump (March 14, 2017)

As I've answered, the reason this is happening is that the root database (in the production Stack Exchange database server) is still called StackExchange.Programmers - a database name that propagates to SEDE and through it to the data dump.

The question here is what to do about it - the current situation is somewhat broken for users of the data dump.

These are the options we have:

  • Do nothing
  • Update the root database name (requires downtime - so very undesirable)
  • Update the update scripts so the SEDE database name is the new name, change will flow to the data dump export (breaks any queries using the old name)
  • Update the export to the data dump so only it gets the new name
  • Something else we haven't thought about

I have added all of these options as answers below - please vote on the one you most prefer, we will try to accommodate the top voted answer.

  • Is PSE the only 'problem'? Are there any other sites with renames? – Patrick Hofman Mar 30 '17 at 14:48
  • As far as I know, yes. But this might come up again in the future. – Oded Mar 30 '17 at 14:49
  • Does leaving the core db as-is cause anyone any problems? (or to be more precise; would leaving the core db as-is and updating the dump/SEDE db cause anyone any problems?) – Cai Mar 30 '17 at 14:58
  • @Cai - leaving the core db StackExchange.Programmers means we have to have a mapping to StackExchange.SoftwareEngineering somewhere in the pipeline. – Oded Mar 30 '17 at 15:00
  • OK, another question... how much downtime? If it's 5 minutes it's probably worth biting the bullet and doing it... if it's 5 hours probably not. – Cai Mar 30 '17 at 15:03
  • 6
  • @Cai - not that simple. There's availability groups and replication to consider, two different data centres etc... – Oded Mar 30 '17 at 15:26
  • 1
    any rough estimate how many SEDE queries are involved that breaks existing queries using the old name? That will only be true for queries where users typed the dbname themselves, right? – rene Mar 30 '17 at 15:54
  • @rene - will have to dig into that, and yes, only queries that reference the db name should be effected. – Oded Mar 30 '17 at 15:56
  • If I rank those in the category power queries created by power users I would simply adapt the scripts to use the new/expected name in the wild and use whatever database name you like on the internal db. That will also prevent down time for this and all type of unforeseen issues. – rene Mar 30 '17 at 16:01
  • Also how did/will you handle the API site name change, just out of curiosity? It appears both programmers and softwareengineering work right now, as synonyms. – Jason C Mar 30 '17 at 16:14
  • 2
    Also another thing to consider, which may actually break a lot more than the DB name change itself, is changing SEDE query link URLs from /programmers/ (which they currently are) to /softwareengineering/. I suppose you could solve that with some 301 magic on data.stackexchange.com. (Or a mass URL replacement in posts, although that wouldn't account for external links.) – Jason C Mar 30 '17 at 16:17
  • 1
  • @FranckDernoncourt That's a different kind of thing. We normally produce the final dump and attach it to the site's proposal in Area 51. Apparently that step got forgotten in this case, but it's a process completely outside the regular quarterly data dump either way at the moment. – Adam Lear Mar 30 '17 at 19:19
  • Hey, so, I'm actually a little confused, now that I think about it. Why is there an issue here? How is a site named "Software Engineering" using a database named Programmers any different than, say, "Motor Vehicle Maintenance & Repair" using a database named Garage and a domain/API name/SEDE URL mechanics? What about "Mi Yodeya" and Judaism? Is there any reason e.g. the database names need to match the site name presented to users? Why is this an issue to think about? Or is the main issue user confusion over the names in the data dump? – Jason C Mar 31 '17 at 3:02
6

Turning some of my comments into an answer:

My gut says leave it as StackExchange.Programmers forever (especially now that it's becoming clearer to me that the main problem is user confusion, in which case, documentation). Is that mapping expensive / risky for future maintenance on SE's end? One problem with changing it is this: If anybody wants to do comparative analysis on multiple archived dumps, they have to be aware of that change. On the other hand that doesn't seem like a common use case. There may be other issues like e.g. rene's question about existing queries.

If that's not possible then my second choice is to bite the bullet and accept the down time and do the rename, no matter how long it takes. Edit: Unless the sole issue here is avoiding user confusion when reading the data dumps, in which case, I doesn't seem like all the effort required to do a rename would really be worth it to avoid a few confused meta posts that could be solved with a readme and a FAQ entry...

I don't think anything in between these two options (do nothing or full, proper conversion and rename) is a good idea. I'm a big fan of all-or-nothing.

Also another thing to consider, which may actually break a lot more than the DB name change itself, is changing SEDE query link URLs from /programmers/ (which they currently are) to /softwareengineering/. I suppose you could solve that with some 301 magic on data.stackexchange.com. (Or a mass URL replacement in posts, although that wouldn't account for external links.)

Just out of curiosity: How will you handle the API site name change, if any? It appears both programmers and softwareengineering work right now, as synonyms. That seems acceptable if it's not an issue to keep it, presuming no new site in the future is created with the name "programmers", of course.


In an effort to help address the primary issue and as a follow-through on my comment here I've put together a list of all sites and their various site-specific "meta" data:

For now it's just in a spreadsheet. This table was automatically generated by combining info from Sites.xml in the data dump, the file name list from the torrent, the API, and sys.databases in SEDE, and contains:

  • Various ID numbers.
  • Dump filenames.
  • Site URLs and aliases.
  • Site API keys.
  • SEDE database names.
  • A couple other things.

I am thinking of how to best present this information in an organized manner e.g. a FAQ entry, given that there's a lot of it and also there is no support for tables in posts.

15

Jeremy already mentioned Beer SE in the comments, but I'm pretty sure we have a few other renames in the past as well. Moderators -> Community Building comes to mind, possibly others. Heck, Mechanics' DB is still StackExchange.Garage. We'd need to do figure out what all needs to be updated.

Given that this isn't likely gonna be the last time we rename a site in a way that affects the subdomain/database names, I think I'd prefer building some sort of aliasing thing into the dump process, so that we don't have to take a site offline every time we need to do a rename.

Or, even simpler, provide release notes with a data dump and/or note these few cases in our description on archive.org.

  • Just out of curiosity, why does changing the title text of a site affect its database name? – Jason C Mar 31 '17 at 3:05
  • 1
    @JasonC The title doesn't, but the subdomain could. By convention, all site DBs are created as StackExchange.<subdomain>. It's obviously not a requirement or else we wouldn't be in this situation where the site name/subdomain have changed but the DB name has not, but there you go. If we were to update the DB name for Software Eng, I feel we'd need to do that for all sites that were similarly renamed before and all that will be renamed in the future, which I also think is excessive. – Adam Lear Mar 31 '17 at 3:07
  • And the main problem with mismatched DB names + site names is... user confusion when reading the data dumps? That's the main reason for this discussion? – Jason C Mar 31 '17 at 3:09
  • 1
    @JasonC As far as I understand it, yes. – Adam Lear Mar 31 '17 at 3:10
  • 1
    Well, heck, in that case, why are we even talking about it? +1000 for just adding a readme to the dumps and/or a community wiki meta post about it (e.g. adding a list here -- I could even do that right now, or as a new faq-proposed post), problem solved. – Jason C Mar 31 '17 at 3:12
  • 1
    I wrote a tool to piece together a bunch of info from the dump, sede, and the api, so now I've got docs.google.com/spreadsheets/d/… -- now I just have to figure out how to present it. – Jason C Apr 1 '17 at 8:44
4

We kinda solved this problem once already... Site Names in stackexchange_files.xml

The goal here was to put the familiar (or at least, current) name of the site in Sites.xml with enough information to map that to the data. So the problem here is either:

  1. Folks don't know how to use this file to establish the right mappings, or
  2. The file contains inaccurate / incomplete data and thus cannot be used for the right mappings

...let's figure out which of these is a problem. Here's the current entry for Software Engineering:

<row Id="22" 
     TinyName="programmers" 
     Name="Software Engineering" 
     LongName="Software Engineering" 
     Url="http://softwareengineering.stackexchange.com/" 
     ImageUrl="https://cdn.sstatic.net/Sites/softwareengineering/img/logo.png" 
     IconUrl="https://cdn.sstatic.net/Sites/softwareengineering/img/icon-16.png" 
     DatabaseName="StackExchange.Programmers" 
     Tagline="Q&A for expert programmers interested in professional discussions on software development" 
     TagCss=".post-tag{
 background-color:#E0EAF1;
 border-bottom:1px solid #3E6D8E;
 border-right:1px solid #7F9FB6;
 color:#3E6D8E;
 font-size:90%;
 line-height:2.4;
 margin:2px 2px 2px 0;
 padding:3px 4px;
 text-decoration:none;
 white-space:nowrap;
 }
 .post-tag:hover {
 background-color:#3E6D8E;
 border-bottom:1px solid #37607D;
 border-right:1px solid #37607D;
 color:#E0EAF1;
 text-decoration:none;}" 
      TotalQuestions="44062" 
      TotalAnswers="136557" 
      TotalUsers="216079" 
      TotalComments="391424" 
      TotalTags="1626" 
      LastPost="2017-03-13T21:40:40.657" 
      BadgeIconUrl="https://cdn.sstatic.net/Sites/softwareengineering/img/apple-touch-icon.png"/>    

Now, that's enough information to render an icon, the proper name, and either resolve or compose canonical URLs for posts on that site. Also enough information to render a stylized tag, on the off-chance that's ever important.

But it lacks sufficient information to identify the file in which the data is stored!

Each site lives in a 7z archive with the name of the site's domain. Some sites - like Software Engineering - use the old domain (programmers.stackexchange.com.7z) which isn't listed in Sites.xml; others - like Motor Vehicle Maintenance and Repair - use the current domain (which can be obtained from the Url field).

Using the database name doesn't work here either; there's no garage.stackexchange.com.7z.

(there are some other head-scratching inconsistencies in this file, like the use of Statistical Analysis as the name for the site that calls itself "Cross Validated" everywhere else... But that's out of scope for this discussion.)

Proposed solution

This would all be a non-issue if we were consistent about using the canonical domain for each site as its archive name. But if we can't do that, adding an "ArchiveDomain" field to Sites.xml that would reliably map to the domain used for these files would suffice.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .