17

It looks like Stack Overflow has been refreshed properly, but the next couple of sites are not making any progress.

SEDE query results

Improve this question
4
  • 3
    definitely something wrong: data.stackexchange.com/stackoverflow/query/1863913
    – rene Mod
    Commented Sep 29 at 13:57
  • 16
    Yes, I'm on it, we had a server failure this week and are working around it, plus I lost power for 15 hours, plus the lawn crew cut my fiber. Please give me today to sort everything out.
    – Aaron Bertrand Staff
    Commented Sep 29 at 14:26
  • 6
    I mean, as for as reasons for delays go... ... ow...
    – Journeyman Geek
    Commented Sep 29 at 15:15
  • 5
    @AaronBertrand well, at least the dog didn't eat your notes. ;-)
    – Shadow Wizard
    Commented Sep 29 at 16:53

1 Answer 1

Reset to default
32

We had a perfect storm of failures that led to a delay in this week's SEDE refresh.

  1. On Thursday, during a standard patching operation, a Colorado secondary (where most of the SEDE data comes from, other than Stack Overflow) didn't came back online. It's still not online but, on Thursday, I had high hopes that it would be operational by Friday.

  2. On Friday at 10:51 UTC, I lost power due to Helene. It took Duke Energy 15 hours to get my power back online. During the day, we learned that the Colorado server was in worse shape than we thought, and that it will still take until at least tomorrow to get remote hands into the cage. I verbally walked a colleague through pointing SEDE at a New York secondary, knowing that this would be much slower to load from the Colorado source, but it should still work.

  3. On Saturday afternoon, just a couple of hours before the SEDE process was scheduled to run, my lawn crew inadvertently cut the fiber cable that AT&T buried too shallow. Since AT&T's first repair appointment was mid-week, I immediately opened a backup account with Spectrum (who I had an account with before), with the hope of being online before the SEDE process ran, so I could oversee it. As it turns out, their original cable was also buried too shallow, and I needed to wait for a rescue tech appointment. Which didn't arrive until...

  4. ...Sunday morning. Spectrum ran a new cable, and I had my service restored by 14:30 UTC. At this point, I began troubleshooting the SEDE process, which had a few misconfigurations because it simply wasn't written to run from a NY secondary. And, as suspected, it is a lot slower.

Sorry for all the background, I just wanted to make it clear that we had a contingency happily in place, but it was sabotaged by cascading failures.

UPDATE Monday at 01:14 UTC:

All databases are now available. This was textbook quick-fix engineering - I changed from a pull to a push model to get things out, and need to circle back for tidying. If you spot any discrepancies, please let me know.

Improve this answer
5
  • 21
    Wow. Firing up service with Spectrum when your AT&T line was cut shows an extreme amount of dedication to getting back to it. Anyone else would've happily waited for the repair service to come out. Thank you for taking your work so seriously. It's sincerely appreciated by the community.
    – Spevacus Mod
    Commented Sep 29 at 22:52
  • 4
    Wanted to reiterate: chat.meta.stackexchange.com/transcript/89?m=9786245#9786245
    – Sonic the Anonymous Hedgehog
    Commented Sep 29 at 23:18
  • 9
    Still one unanswered burning question remains: How is your lawn?
    – rene Mod
    Commented Sep 30 at 7:25
  • 4
    @rene Meh, it's almost a single shade of green.
    – Aaron Bertrand Staff
    Commented Sep 30 at 16:30
  • 3
    Thanks for being amazing, Aaron. :)
    – V2Blast
    Commented Sep 30 at 20:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .