21

Normally, when you click the Inbox or the Achievements icon in the top bar, the notification (x new messages, +y reputation) clear. They still do, but upon refreshing the page, they're back to their old values.

enter image description here

It seems the counters have been reset a few minutes ago (my 'score' was 5 new messages and somewhere North of 100 reputation gained), but now they're back to a persistent 1 new message and +15 reputation. Unlike @Machavity I'm experiencing this on all sites, not just the ones which were down for maintenance earlier today.

Improve this question
6
  • 1
    Yup. And chat pings are really delayed. I usually get them within a minute and they're taking much longer, if they ever turn up.
    – Catija
    Commented Apr 25, 2019 at 13:42
  • They clear for me everywhere BUT Meta.SE and Meta.SO, which is what went down earlier
    – Machavity
    Commented Apr 25, 2019 at 14:08
  • @Machavity strange, I'm experiencing it on all other sites as well (I checked half a dozen).
    – Glorfindel Mod
    Commented Apr 25, 2019 at 14:11
  • I'm also getting it on all sites I visit.
    – nitsua60
    Commented Apr 25, 2019 at 14:18
  • Ok, that was weird. Was on Space.SE with 0 notices showing when I had a notice come in and it said 3 (which is what I get on Meta.SE), not 1. Possibly related
    – Machavity
    Commented Apr 25, 2019 at 14:20
  • Repro'd on MSO and SO
    – Zoe - Save the data dump
    Commented Apr 25, 2019 at 14:24

1 Answer 1

Reset to default
27

This was an issue with a library update we just pushed out which seemed fine but depending on assembly load order created a race that exposed a binding redirect issue with System.Numerics.Vectors because the .NET assembly loader is just oh so fun.

The net result was this library which gets a first or second hit between Providence and StackExchange.Redis, when the latter was a binding victim (because it tried to load an older version, 4.1.3, which wasn't there and wasn't redirected). The build produced the latest 4.1.4 so when redis tried to load it had the secondary impact of falling back to a 5 second timeout cap. But, since we connect to many connections and secondaries for redundancy, this resulting in a 100+ second app startup stall (or a chance to have one rather).

DLL diff

In the end, this resulted in a terribly long startup time and still no redis at the end (which is what caused this issue), because the socket pipeline never started up.

The depth of library ref changes (transitive dependencies) here presented no issues or warnings for binding redirects in tooling (VS has a built-in warning for this...but it doesn't trigger) and only presented in prod and only sometimes. We deployed 2 tiers of servers and it wasn't until a follow-up build 20 minutes later that this issue started cropping up on app restarts after the old app domains evacuated.

Binding redirects are the devil. So we're moving to .NET Core where this crap doesn't happen.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .