Oops. My Bad. An import is running now!
We have been doing several database tier upgrades over the past month that are almost completed. I view my job as being that you don't know about them unless we tell you, obviously that didn't happen here though. Here's what's happened recently:
SQL Cluster #1:
- Single 2TB P3700 PCIe NVMe SSD installed in each node
- Stack Overflow moved to this drive, all previous databases on the existing 24x 200GB Intel 710 RAID 10 array.
SQL Cluster #2:
What broken here was the SQL Cluster #2 replacement. To avoid downtime many things were shuffled behind the scenes here, many of these PCIe cards have been in more than 1 server as we move things around to do upgrades behind the scenes as much as possible. For cluster #2 what we did was replace the auxiliary node in the NY data center requiring only one ~20 second outage while we moved traffic over. In Oregon though, being a DR data center for this cluster at the moment made things simpler and we did not replace OR-SQL02 but instead added OR-SQL03 (which matches our internal parallel service naming scheme).
We ran with 2 replicas in Oregon for a while after the addition to make sure nothing pointed to OR-SQL02 when we shut it down (fail!), then I finally shut down OR-SQL02. This broke SEDE since it was linked directly to OR-SQL02 not an availability group listener for data. I have just repointed it at OR-SQL03 and it's running again but it will take about 2 hours to finish.
Why don't we use an availability group listener over there? We can't because a listener is only up on the current master, which is almost never Oregon. We are working with Microsoft to improve the HA solutions out there given the current gaps with AGs in minority sites.
If anything of this doesn't make sense - comment and I'll try to clarify.