On Saturday, January 22nd, we had a failure on a backup share that won't be fully resolved until Monday.
We've immediately re-routed backups to multiple secondary locations, so that concern has been mitigated.
Unfortunately, the Data Explorer refresh job and the processes that feed into it were disabled as a precaution.
We had two options:
- Cancel the SEDE refresh job and wait until the underlying issue is resolved
- Dig into the processes to determine how to make them piece together backups from different locations
Due to it being the weekend, we decided that option #1 was the only feasible one, and hoped to refresh Data Explorer as soon as everything was back in order. Expectation was Monday at the earliest and hopefully no later than Tuesday.
Option #2 wasn't something I felt comfortable loading onto probably multiple people as a surprise weekend project, a mere few hours before the job was scheduled to run.
This precaution turned out to be unnecessary, as my digging into the process yielded today and as my answer below explains.