8

The current data migration is about 8 hours over due for the 3:00AM UTC (approx) time it is normally done.

Just wondering if there's a problem?

Update:

OK, there's definitely a problem. The migration is now (UTC 10AM Monday) 1 day 7+ hours overdue (or by those who believe that Monday is the day, 7+ hours overdue - either way, it's late).

Improve this question
12
  • "The data is updated early every Monday morning" - I don't think it's Monday already. – Shadow Wizard is Vaccinating Jan 11 '15 at 12:47
  • possible duplicate of When will data.SE get updated data? – Shadow Wizard is Vaccinating Jan 11 '15 at 12:48
  • 2
    @ShadowWizard no. It's updated every Sunday morning. At least for stackoverflow data - i can assure you. – Bohemian Jan 11 '15 at 12:49
  • 1
    Probably just a bonus, but I wouldn't start worrying at least one full day after the officially written time, which is Monday. – Shadow Wizard is Vaccinating Jan 11 '15 at 12:50
  • 1
    @shadow OK, perhaps a bonus, but it's been like clockwork for ages. It's not a big deal to me, just putting it out there. Last time I posted a similar question there was a problem no one seemed to be aware of. – Bohemian Jan 11 '15 at 12:52
  • 1
    Might be true, but pinging a dev in chat sounds better than starting a bug report. – Shadow Wizard is Vaccinating Jan 11 '15 at 13:06
  • 4
    @ShadowWizard: for months now the normal update time has been Sunday early morning UTC. I think a bug report here is just as appropriate. – Martijn Pieters Jan 11 '15 at 16:16
  • 1
    Wild guess: SQL cluster upgrade on January 6 broke something. – user259867 Jan 11 '15 at 16:18
  • correct that for like most, if not all of 2014, it updated early morning UTC on Sunday. related meta.stackexchange.com/questions/234390/… – CRABOLO Jan 12 '15 at 0:14
  • 3
    FYI - Migration still hasn't run as at this message – Bohemian Jan 12 '15 at 1:49
  • 4
    As far as I am aware, in the past year, there has been only one irregularity or delay in the SEDE update, and that was when an additional update was done part-way through a week to include Japanese.se data in to the system. Although the FAQ may say the update happens on Monday mornings, it is clearly wrong, and indicates that the FAQ needs updating as much as the data ;-) – rolfl Jan 12 '15 at 3:25
  • 3
    in a very C3PO moment, if I may quote Nick: "oh yeah I guess it wouldn't have seeing as I shut down the source server" – Marc Gravell♦ Jan 12 '15 at 14:18
10

Oops. My Bad. An import is running now!

We have been doing several database tier upgrades over the past month that are almost completed. I view my job as being that you don't know about them unless we tell you, obviously that didn't happen here though. Here's what's happened recently:

SQL Cluster #1:

  • Single 2TB P3700 PCIe NVMe SSD installed in each node
  • Stack Overflow moved to this drive, all previous databases on the existing 24x 200GB Intel 710 RAID 10 array.

SQL Cluster #2:

What broken here was the SQL Cluster #2 replacement. To avoid downtime many things were shuffled behind the scenes here, many of these PCIe cards have been in more than 1 server as we move things around to do upgrades behind the scenes as much as possible. For cluster #2 what we did was replace the auxiliary node in the NY data center requiring only one ~20 second outage while we moved traffic over. In Oregon though, being a DR data center for this cluster at the moment made things simpler and we did not replace OR-SQL02 but instead added OR-SQL03 (which matches our internal parallel service naming scheme).

We ran with 2 replicas in Oregon for a while after the addition to make sure nothing pointed to OR-SQL02 when we shut it down (fail!), then I finally shut down OR-SQL02. This broke SEDE since it was linked directly to OR-SQL02 not an availability group listener for data. I have just repointed it at OR-SQL03 and it's running again but it will take about 2 hours to finish.

Why don't we use an availability group listener over there? We can't because a listener is only up on the current master, which is almost never Oregon. We are working with Microsoft to improve the HA solutions out there given the current gaps with AGs in minority sites.

If anything of this doesn't make sense - comment and I'll try to clarify.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .