34

On Saturday, January 22nd, we had a failure on a backup share that won't be fully resolved until Monday.

We've immediately re-routed backups to multiple secondary locations, so that concern has been mitigated.

Unfortunately, the Data Explorer refresh job and the processes that feed into it were disabled as a precaution.

We had two options:

  1. Cancel the SEDE refresh job and wait until the underlying issue is resolved
  2. Dig into the processes to determine how to make them piece together backups from different locations

Due to it being the weekend, we decided that option #1 was the only feasible one, and hoped to refresh Data Explorer as soon as everything was back in order. Expectation was Monday at the earliest and hopefully no later than Tuesday.

Option #2 wasn't something I felt comfortable loading onto probably multiple people as a surprise weekend project, a mere few hours before the job was scheduled to run.

This precaution turned out to be unnecessary, as my digging into the process yielded today and as my answer below explains.

Improve this question
2
  • 18
    A totally reasonable (and well-reasoned) decision. This announcement is a fantastic model for all announcements from staff, honestly disclosing the problem(s), the options, and the thought process that was involved in making the decision. Thank you for that!
    – Cody Gray
    Jan 22 at 4:39
  • 7
    That's the first time ever in SE history that staff answer comes before the bug report. Always, but always, the developers waited until there was a bug report, and after few hours (in case of major issue like SEDE not updating) would post an answer. So, I second Cody's thanks. :)
    – Shadow The Spring Wizard
    Jan 22 at 7:55

1 Answer 1

Reset to default
19

The weekly SEDE refresh was started about 12 hours later on Sunday, January 23rd, at 14:40 UTC and finished in 8h 24m (a little after 23:00 UTC).

A bit of a mea culpa: as I dug into the refresh job today to see if I could tinker with where it looks for backup files, I realized that it actually doesn't rely on the backup files directly, and that there is no effort required to make this process more tolerant.

SEDE should be all refreshed with last week's data by about 23:00 UTC, including some fresher data from overnight while the job was disabled.

Sorry for the delay, the interruption, and the slightly misguided panic.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .