I apologize for today (well I guess yesterday) being far less than awesome. As Architecture lead and main author of this set of changes, it's on me.
We deployed some massive changes that we knew were risky but deployed carefully and up until the last build of the last application they looked awesome. In a "I wish more things would break so I don't think I'm crazy" level of awesome.
Then we built
stackexchange.com (the app we considered the least likely to break anything). That's when shit hit the fan. Queues shot up. Alarms went off. Queries slowed down. Our SQL Servers slammed against 20Gb network throughput limits. HTTP requests piled up. We bounced between offline and unstable for 20 minutes. Then it went downhill from there.
The aggregator (the thing that powers the top bar) was one of several pieces of infrastructure that took severe collateral damage in the process. The aggregator is very old and in need of a rewrite. Tonight a large chunk was rewritten. When we observed a severe performance degradation (only observable at production scale) after the EF Core port resulting in the aggregator being unable to keep up with incoming aggregator events, I rewrote all of the persistence in the aggregator to use Dapper instead. With help from Geoff and Jarrod this is now deployed to production and running smoother and faster.
What this doesn't solve is gaps from failures in the serialization due to order of events queued in the interim with bad IDs on the primary key fields (an ordering error in the EF Core conversion - and my fault). You may notice a few missing events in the aggregator of any type: activity, reputation, posts, etc. We will be fixing those early next week with a full backfill (something we've never attempted and have to figure out and write).
Thank you for being patient as we make some huge codebase changes on the way to .NET Core. It's never our intent to be less than awesome for you, and I hate it when we fall down. When we fail, I take it very personally. I love that the community is understanding when things go wrong - you are the most important thing to all of us here. <3