33

Since at least 2 hours ago (7:00PM (EST) on Saturday, February 24, 2024) and about 6 hours past the time today’s maintenance was declared “COMPLETED” by staff member Dalmarus I have experienced login issues.

Not issues logging into the site, but one second I will be logged in… And the next second I am seeming not logged in. Reloading the page seems to help or caused it; reloading the page seems to toggle this behavior.

For example, here I am trying to load my inbox from a supposedly “logged in” page that shows a “Sign up or log in” message:

enter image description here

Connected directly to whatever ISP my device uses: No VPN or TOR or any middle man type of connection; 100% pure connection via the ISP.

The same issue was reported by at least two other users in the comments thread of today’s maintenance post.

enter image description here

12
  • 6
    Same is happening in CV. Reloading seems to work but the issue persists as of now. Commented Feb 25 at 3:35
  • 7
    AskUbuntu is also absolutely weird for me, the "read-only" banner had been regularly shown up on various pages even after I managed to log in. However, most of the time, I cannot even log in, because the page shows: "Login is not available" I have just now taken this screenshot: i.sstatic.net/1CwBP.jpg At the same time, this site here, meta.SE, always works flawlessly.
    – Levente
    Commented Feb 25 at 3:38
  • 1
    @Giacomo1968, Cross Validated :-) Commented Feb 25 at 3:43
  • 1
    Confirmed - Questions on Bicycles.SE are sometimes showing the Read Only header, or if not a vote will show a "we're in read only mode" message.
    – Criggie
    Commented Feb 25 at 7:14
  • Exact same problems here on U&L_SE. I would just add to the list some minor additional problem : user reputation leagues pages just failing to load correctly.
    – MC68020
    Commented Feb 25 at 9:04
  • 1
    @AaronBertrand : No VPN, no Tor as far as I am concerned.and no known caching proxy as well.
    – MC68020
    Commented Feb 25 at 9:06
  • 1
    @AaronBertrand I don't use Tor, nor a VPN, nor a proxy. As far as I know, my ISP doesn't have a caching proxy, either. I got the issue and refreshing clears it up.
    – VLAZ
    Commented Feb 25 at 9:07
  • 1
    @MC68020 the issue about leagues and avatars was reported (here and here) long before the maintenance began (at 2023-02-24 14:00Z), so I don't believe it's related.
    – Aaron Bertrand Staff
    Commented Feb 25 at 9:13
  • 1
    I've re-issued the set of commands that force all sites out of read-only mode, in the off chance that yesterday's actions didn't make it to one of the application servers. I don't have the ability to check if that's the case, and it is the middle of the night on a Sunday for us, so I just went ahead and forced it. If that was the issue, I'm also struggling to understand what (aside from the things I've mentioned) would cause a few of you to be bounced around by the load balancer.
    – Aaron Bertrand Staff
    Commented Feb 25 at 9:20
  • 1
    @AaronBertrand The problem I was experiencing has now been fixed. Commented Feb 25 at 9:25
  • @AaronBertrand : OK for me here (U&L_SE). Forcing sites out of read-only mode appears to have been efficient since even the orange stripe warning site in read only mode on top of user-reputation leagues is no longer displayed. (user reputation leagues still don't refresh correctly though, must be, as you had asserted here-above another issue.) Thanks and good luck to you in understanding what actually happened.
    – MC68020
    Commented Feb 25 at 9:33
  • 8
    I don’t know that the issue was solved, and I think it’s too early to declare victory. I want to wait to see if we get any reports of similar behavior, hear from everyone who posted here that they’re no longer seeing it, and confirm with SREs at Stack that what I think was the problem was the problem, and that what I think fixed the issue actually did. I’m not confident that will all happen on a Sunday, especially as I was up half the night investigating. Also, I already confirmed that the kubernetes maintenance incident is completely unrelated to this platform. That one I am 100% sure of.
    – Aaron Bertrand Staff
    Commented Feb 25 at 14:18

2 Answers 2

12

By all things we can measure, we believe this issue has been resolved.

What happened? (Working theory.)

The command that moves our sites out of maintenance mode did not complete successfully on one of the several web servers that might serve you a page. When you were unlucky enough to hit that web server, you got the read-only banner, or a message stating you weren't logged in. We ran the command again and voila.

Want more detail?

Okay, but some necessary background first.

Like most even moderately complex web applications, there are several tiers that help satisfy the volume of traffic and data we serve. Not surprisingly, the architecture looks something like this (grossly simplified, and doesn't include other obvious things you already know we use, like Cloudflare, Elasticsearch, Redis, APIs, and so on):

basic web app architecture on a good day

We have databases that accept and supply data from web servers; your request to view a page comes through load balancers and gets directed to one of those web servers. The web servers have application pools and what you might call a "sites cache" - among other things, it contains the list of sites, their URLs, and current status (read-write or read-only). This cache doesn't get refreshed often, since our list of sites is relatively static.

Like any architecture, the servers at any layer of this tier occasionally require maintenance of various types. To protect you from losing data (for example, a server transitions state or goes offline at the moment you're writing), we proactively go into read-only mode.

There is more to it, but you can think of read-only mode as simply updating the sites cache for any given (or all) site(s), to effectively say, "this site is read-only." That's what puts the banner on the page, prevents you from logging in, and returns a message if you try to vote or post - even if you're already logged in. I won't draw it again, but picture the above diagram with orange "cache" boxes that say "read only."

We think this is a better compromise than just presenting a "down for maintenance" page, which can happen for "site down" or service degradation incidents. It also lets us put individual sites in read-only mode, which is why this weekend Stack Overflow was only read-only for a couple of minutes, and the rest of the network for only about 10 minutes. For most of you.

We have commands we can run (manually or more automated) to switch in and out of read-only mode. There are actually three of them we couple together:

  • set read-only on / off
  • reset connections
  • reload the sites cache

At the end of this maintenance, we ran these commands to turn read-only off. We had a healthy list of sites we manually checked before declaring the maintenance complete:

  • "Is the banner gone?"
  • "Can we post, vote, and comment?"
  • "Is it the same for mobile and not on our VPN?"
  • ...and so on.

Everything looked good.

Later, I saw a few reports like this one. Initially I thought, did the command get reversed? But that couldn't be the case, or this question couldn't have been posted (and I would also see the symptom). I speculated publicly about Tor, VPN, caching proxies, but also spent several hours trying to reproduce the problem - and couldn't.

I ran the three commands again, and several people reported here that the problems went away.

Great, but why?

Ok, now the theory

After several discussions, we believe that what happened is the "reload sites cache" command on one of the web servers did not complete successfully. This made that one web server think that all sites were still read-only:

one web server is grumpy

If you were unlucky, that web server on the right is the one you got.

To check if that was the issue - which I couldn't confirm immediately, since validating the absence of a problem can be challenging - I ran all three commands again, to be thorough. Sure enough, this definitely appears to have solved it.

I'm sorry that this happened and I'm sorry for the delay in providing an answer. But not being an "app tier guy," it's not something I could assert with a decent level of confidence without confirmation from others. After all, it could turn out to be due to other reasons - such as the load balancer or another layer detecting a larger problem and automatically routing requests to our secondary data center, which can only serve read-only traffic (as Josh describes here).

What can we do to be better in the future?

  1. For a start, we can add an additional step to the "come out of maintenance" process:

    • set read-only off
    • reset connections
    • reload the sites cache
    • restart the application pools, one by one (optional)

    This additional step - if reloading the sites cache doesn't work - will force each application pool to completely reload the current sites cache from scratch.

  2. I couldn't reproduce the symptom (not for lack of trying) probably because I never hit the grumpy/unlucky server. To remedy this, we can invest in a better way to test our set of sample sites (or perhaps all sites, if it can be done efficiently) and hit each one on every web server. This will ensure that if any individual web server thinks it is read-only, we'll see that immediately.

  3. We can add response header information to indicate if a request is read-only. This can make it much easier to detect this traffic in real- or near-real-time in our logs and monitoring tools.

Other notes

  • We have extensive logging from various parts of our architecture that might provide additional insight, but that collection and analysis takes time and, even then, there's no guarantee we'll get a clearer picture than the one we've already formed. This is on my queue.

  • The symptom has nothing to do with the actual maintenance performed, just the apparent failure of the "come out of maintenance" commands - which could have happened for any type of maintenance.

  • To address comments here and elsewhere about what's currently showing on the status page:

    1. I want to assure you (again) that the maintenance we were performing on Saturday had absolutely nothing to do with the ongoing maintenance the status page currently shows (labeled "FBB - Adjust Kubernetes Node Pools"). That is for our hosted and non-public Stack Overflow for Teams offering, which will be clearer if you click through to the details. This network doesn't use Kubernetes at all, and doesn't even live in the same data centers as Stack Overflow for Teams.
    2. While it has been the case for several recent maintenance windows, I think it's confusing that they disappear from the status page once they're resolved. I think this treats them too much like an emergency and not enough like a planned outage, or maybe vice-versa, and am requesting this to be changed. This would make each maintenance more of a permanent record correlating to an announcement here on meta, and will better distinguish different maintenance events. But, as always, asking for it doesn't mean it will happen.
2
  • 2
    One of the best answers I've ever seen. Thanks, and keep up the awesome job! Commented Feb 26 at 20:56
  • "since validating the absence of a problem can be challenging" Especially when you are not 100% confident in the fix you have implemented, I feel you. Thanks for this message, this is a great answer! Especially appreciate the "y'all" simplification, now I understand ;)
    – Laf
    Commented Feb 26 at 22:39
5

I think there's some problem with cookies. If I open the SFF site from my bookmark, I get:

Enter image description here

If I press Ctrl + R to refresh, I get:

Enter image description here

Note that the search expression has changed even all I did was refresh the page. Then if I keep hitting Ctrl + R, after a few refreshes, I get:

Enter image description here

It looks like the problem has something to do with local storage, which I assume means cookies.

2
  • 10
    You shouldn't randomly be logged in or out over multiple refreshes due to cookies. This more likely indicates a bad node you connect to but the load balancer bounces you around different ones only occasionally landing on the bad one.
    – VLAZ
    Commented Feb 25 at 8:22
  • 3
    OK. In any case it seems to have been fixed now. Commented Feb 25 at 9:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .