9

At about 7:55 EDT 23 OCT, attempting to refresh the main Stack Overflow page threw an error, and was in read-only mode when refreshed a few seconds later. The status page references a maintenance that should have been completed about ten hours previously. It's been down for almost an hour now, with no update, no ability to sign in, and when I try to apply my standard question filter (which is long), it throws an error.

What's up?

Improve this question
13
  • Did you try a hard refresh, just in case?
    – VLAZ
    Commented Oct 23 at 12:50
  • @VLAZ I did a hard refresh... why, can you access the site(s)?
    – Gordon Brinkmann
    Commented Oct 23 at 12:53
  • I can at this time.
    – W.O.
    Commented Oct 23 at 12:53
  • @GordonBrinkmann yep. I've not experienced any downtime. But there was some maintenance this night, might be a stale load balancer or something that served the maintenance page and that was cached.
    – VLAZ
    Commented Oct 23 at 12:54
  • @VLAZ I just realized I only have this problem with Chrome. Edge and Firefox have no problem. Weird.
    – Gordon Brinkmann
    Commented Oct 23 at 12:56
  • Hard refresh didn't do anything for me; I had to delete all SO cookies before it would come out of R/O. Still have NFC why it horked up the hairball in the first place...
    – Jeff Zeitlin
    Commented Oct 23 at 13:36
  • 2
    Seems like this might be the same problem that's been reoccurring for 5 months now. Initially reported in meta.stackexchange.com/q/399904 . Also was reported in meta.stackexchange.com/q/403014 meta.stackexchange.com/q/402675 and meta.stackexchange.com/q/401833 .
    – Spevacus Mod
    Commented Oct 23 at 13:41
  • 9
    There's been an ongoing issue since the company moved to Cloudflare, where some folks get routed to the read-only copy of the website. My guess is that a small percentage of folks keep getting routed to their DR data center, and they just can't figure out why. I just kinda wish the company would be transparent about what's happening since it's been like this for MONTHS.
    – AMtwo
    Commented Oct 23 at 13:42
  • I got this as well, only on one of the 3 machines I have here and only for Stack Overflow. Clearing cookies fixd it.
    – greg-449
    Commented Oct 23 at 14:02
  • I posted an answer to AskDifferent around 8:30am US EDT and the answer has completely vanished, no trace of it. I don't recall seeing any error etc when submitting. Strange, not sure if it's related...
    – luckman212
    Commented Oct 23 at 15:41
  • Getting this on Writing, but I'm not sure what to try. I cleared the recent cookies, which logged me out of all other SE sites, but the problem wasn't fixed (iOS Chrome).
    – Laurel
    Commented Oct 23 at 17:20
  • 2
    @Laurel You might want to contact [email protected] -- I know that the company is aware of this issue, but not sure why it's been ongoing for so many months... but I think your only options are to wait and see if it eventually gets better, or contact them directly for help./
    – AMtwo
    Commented Oct 23 at 18:33
  • This happened to me today on Mathematics Stack Exchange. It seems that the issue is still not resolved
    – user1301875
    Commented Nov 1 at 21:20

1 Answer 1

Reset to default
8

TL;DR: We tried something new this maintenance that allowed us to keep the site fully functioning for most of the maintenance. It was working well until we hit a undiscovered bug in our code during cleanup that caused large parts of the site to go into read only mode.

Long version: With the switch to Cloudflare, we're able to dynamically steer traffic to our secondary site that runs in read only mode if the primary site goes down. That way, if something happens to the primary site, users can still access the Stack Overflow and SENetwork sites albeit in a slightly degraded state. For this planned maintenance, instead of directing all traffic to the read only secondary site for the duration of the maintenance, we decided to lean on the dynamic steering so that we could reduce the time the site was in read only mode to a minimal amount. This worked and only some traffic was sent to the read only secondary for a brief period of time.

The problem actually happened when we were finished with the maintenance. We marked the maintenance as completed and started to clean up outstanding configurations that we had prepared ahead of time for the maintenance but ended up not needing. A bug in our code caused our DNS to force traffic to the read only secondary site unintentionally. Due to the issue being an undiscovered bug, it took some time for us to find the culprit and correct the problem. Another factor is we had already marked the maintenance as completed and forgot to re-open it so the status page stopped getting updated for that particular incident. We've since fixed the bug and did an audit our code to make sure this won't happen again.

DNS Haiku

Improve this answer
6
  • 1
    Can something be done to prevent browsers from caching the DNS entry to the failover server? We frequently see problems where people are still seeing the site in read-only mode long after maintenance concludes, because their browser has cached the DNS lookup to the failover datacenter.
    – Sonic the Anonymous Hedgehog
    Commented Oct 24 at 19:04
  • 2
    We discovered this also and will be looking into it to see what we can do about it.
    – Josh Zhang StaffMod
    Commented Oct 24 at 19:18
  • The "original" report for what @Sonic said is this one, with answer being "clear cache", which does work for most. So perhaps if you're aware, please throw a status-review on it?
    – Shadow Wizard
    Commented Oct 24 at 19:50
  • 2
    @ShadowWizard that needs to get triaged through the CM team first, if I change the status tag it might get lost in the fray because there wouldn't be an internal ticket tracking its progress.
    – Josh Zhang StaffMod
    Commented Oct 24 at 20:02
  • 2
    @ShadowWizard Status-review tag added (along with a quick answer pointing here and to the original maintenance post). I'll also link the internal tickets for Josh and crew working on it.
    – Dalmarus StaffMod
    Commented Oct 24 at 20:38
  • Awesome, thanks!
    – Shadow Wizard
    Commented Oct 25 at 16:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .