87

Update: The maintenance went as planned.

tl;dr; Planned service interruption that will impact Stack Exchange Data Explorer (SEDE). The site will be inaccessible for up to 12 hours on February 5, 2020 starting at 2:00pm UTC.

Short Version:

There will be an extended service outage for up to 12 hours starting at 2pm UTC (9am US/Eastern) on February 5, 2020. During this time only the Stack Exchange Data Explorer will be inaccessible. This maintenance will not impact Stack Overflow, the Stack Exchange sites, Jobs, Chat, Teams, or any Enterprise cloud hosted instances.

Longer Version

What we'll be doing

Currently, the SQL Server for the Data Explorer is running Windows Server 2012 R2 and SQL Server 2017. I am slowly upgrading our servers to SQL Server 2019, and in order to upgrade the server for SEDE, I need to upgrade the operating system. During this maintenance window, I will be doing the following:

  1. Upgrading the operating system to Windows Server 2016
  2. Installing SQL Server 2019
  3. Setting up the databases for querying which might involve having to refresh the databases (a lengthy process)

Yes, the outage window is long. This is because I’m baking in some buffer time for issues that might come up, including the need to refresh the databases. Typically the databases are refreshed every Sunday and it takes about 7 hours to complete. I’m hoping we won’t have to refresh the databases after steps 1 and 2, but I won’t know until after I get SQL Server 2019 installed and can confirm that the databases survived the maintenance without issue.

Questions or concerns?

Please post a comment or answer below; I’ll do my best to address any concerns between now and the maintenance window.

Improve this question
22
  • 41
    Let's hope it won't be as stressful as the last migration / upgrade you did. Good luck!
    – rene
    Jan 28, 2020 at 16:53
  • 6
    @rene Let's hope. :)
    – Taryn
    Jan 28, 2020 at 16:54
  • 3
    wish you success for upgrading that. I hope it'll be more smooth then the last SQL server upgrade saga, from the blog post you wrote that seemed a tad stressful.
    – Magisch
    Jan 28, 2020 at 17:03
  • 2
    @rene Not separately, no. Our internal instance and it's database is running on a server that is already at an OS that will work with SQL Server 2019. That will be just an upgrade of SQL, when I get to it.
    – Taryn
    Jan 28, 2020 at 17:04
  • 5
    "I’m hoping we won’t have to refresh the databases after steps 1 and 2" - you haven't done a dry run of this on a non-production instance?
    – Jamiec
    Jan 28, 2020 at 17:16
  • 17
    @Jamiec Yes, we have done this before, but there are always possible issues with these types of installs.
    – Taryn
    Jan 28, 2020 at 17:18
  • 3
    @Jamiec youve never encountered an issue in production that didnt come up in testing?
    – WhatsThePoint
    Jan 28, 2020 at 17:31
  • 18
    @Jamiec To be clear, I have plenty of experience breaking our servers while attempting to upgrade them.
    – Taryn
    Jan 28, 2020 at 17:37
  • 3
    I didnt mean to imply otherwise, I promise.
    – Jamiec
    Jan 28, 2020 at 17:40
  • 7
    Upvoted for including the name of the day, Wednesday, in the title. The inclusion is a useful mnemonic, and I'm glad you remembered from last time.
    – Rob
    Jan 28, 2020 at 17:43
  • 6
    @JMP Doing what? Upgrading to SQL Server 2019? We started with out test servers, and then moved to dev. Now, I'm working on other servers that aren't a part of the main SQL clusters. Lastly, I'll tackle the main clusters, but I tend to move from less critical environments to more critical to hopefully find issues with the install process, etc. I'm targeting everything on SQL Server 2019 by the end of March 2020, so it's coming, but it's a gradual process.
    – Taryn
    Jan 28, 2020 at 19:13
  • 3
    Any reason to pick Windows Server 2016 over Windows Server 2019?
    – Ismael Miguel
    Jan 28, 2020 at 22:16
  • 10
    @IsmaelMiguel Yes, we encountered a lot of issues when we attempted to upgrade to Windows Server 2019 on our main clusters last year. As a result, we aren't comfortable going that route yet. I documented some of those issues on my blog, if you're curious.
    – Taryn
    Jan 28, 2020 at 22:22
  • 2
    Twelve hours? Remember to take two packed lunches then.
    – Andrew Morton
    Jan 29, 2020 at 9:42
  • 4
    Hope all goes well and none of this happens: xkcd.com/2261
    – Bryan Krause
    Jan 29, 2020 at 22:41

4 Answers 4

Reset to default
12

Would it be possible to also fix Psychology & Neuroscience's broken icon links? They're the icons for Cognitive Sciences + Cognitive Sciences Meta on the main Data Explorer page.

Improve this answer
2
  • 14
    I am aware of the request to fix that issue but it’s not as easy as it first appears. That fix is out of scope for this maintenance window. I’ll circle back to it after we’re done and see if we can resolve it.
    – Taryn
    Jan 29, 2020 at 3:05
  • 7
    There is a full report here: meta.stackexchange.com/a/332277/158100 with a proposed fix.
    – rene
    Jan 29, 2020 at 7:38
10

Are you planning to increase the database compatibility level? I'm not sure if the promised performance improvements have any effect on a typical SEDE query, but it (in my limited experience with databases) it never hurts.

Improve this answer
2
  • 11
    We will increase the database compatibility level, but probably not on the day of the upgrade. I want to be sure we don't hit any issues with any of the other changes before adding a compatibility level change as well.
    – Taryn
    Jan 29, 2020 at 12:09
  • @Taryn that makes sense, thanks!
    – Glorfindel Mod
    Jan 29, 2020 at 16:54
6
  1. Do you have a rollback plan should the upgrade to 2016 fail?
  2. If you do encounter a problem during the upgrade process that may take longer to fix than the change window allows, will you initiate the rollback plan to prevent from breaching the provided change period?
  3. If the plan changes and the change window will be breached, where will the notices and updates be posted advising users of the extended maintenance?
Improve this answer
3
  • 39
    Are you that depended on SEDE?
    – rene
    Jan 29, 2020 at 7:36
  • 11
    1) If needed, we could move back to Windows 2012 R2. 2) Ideally, no. Unless the problem is impossible to solve quickly, we would probably push to get it fixed even if that means it takes longer than the proposed change window. 3) We tend to post updates to @StackStatus as we're working or during outages. If things take longer than expected, I will post updates there.
    – Taryn
    Jan 29, 2020 at 12:24
  • @rene Not dependant on anything. I was more curious about the change process and how a potential rollback would be handled. You learn a lot about an IT team in how they approach production upgrades. Whether they have a rollback plan, when they know to cut their losses and initiate it, how they communicate possible problems to end users etc. A team without a rollback plan and does not communicate issues encountered to end users are just a faceless IT cost centre rather than a value add.
    – Drew
    Jan 29, 2020 at 22:26
4

Adding to @Rebecca's answer, the Writers SE icon also seems to be broken.

Also Writers Meta, And Cognitive Sciences Meta.

Thanks to @rene for linking:

A site URL changed and now its icon in SEDE is broken

Improve this answer
2
  • 4
    Complete bug report here: meta.stackexchange.com/a/332277/158100
    – rene
    Jan 29, 2020 at 8:03
  • 2
    As I mentioned under Rebecca's answer, this is out of scope for this maintenance window. I'll check back after we're doing with all the upgrades to see if we can implement a fix.
    – Taryn
    Jan 29, 2020 at 12:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .