Tracking this down was driving me absolutely insane, but a fix is rolling out now.
Here's the fix:
//GetCache().Set(key, DateTime.UtcNow, OneMonthInSeconds);
GetCache().Set(key, DateTime.UtcNow, OneMonthInSeconds, broadcastRefresh: true);
I'll try and explain what was happening since this took so long to get a fix out for. The cause didn't occur to me until reading the code for the 200th time just now. Here's what happens when you open the achievements dropdown:
- Your browser hits
- That route sets 3 values:
- Integer: Unread Rep to 0
- Integer: Unread Badges to 0
- DateTime?: Your last read date, now:
Here's what happens when your rep is recalculated (pretty much this only happens when a post you possibly had any stake in is deleted or undeleted):
- Your reputation is recalculated on the site that triggered a recalc
- Your RepHistory is updated on that site
- An aggregator event is added to a redis queue for a network recalc to happen
- An aggregator instance (whichever web server currently won the election) processes that item
- Your NetworkRepHistory (aggregate from all sites) is synced up by doing a diff with the site that triggered the recalc
- Your "last read" date is fetched from cache
- A query is run to total up your actual network rep change between that date and now
- We update your total unread rep and send it realtime
Ok, so here's where that went south. In our caching layer, integer gets, sets, and increments are atomic and go directly to redis. The
DateTime? fetch does not, it goes through L1 cache which is HTTP Runtime cache on that web server. That means you hit one web server & app pool to set your "last read" date on achievements while the process that was fetching your total after a recalc was hitting another, almost guaranteed to be out of date (up to a month, ouch).
I pushed the change above (
broadcastRefresh: true) which invokes a cache clearing mechanism we have for these edge cases that are very apparent to the user when cache decay occurs. It uses the redis pub/sub mechanisms to inform other applications that they need to clear their L1 cache of that key. This means when the aggregator runs it'll get a fresh brand-spankin-new date for the total query that is, most importantly, accurate.
I really am sorry this bug lingered so long, it was subtle and not apparent to a few of us trying to figure this issue out. Still, it was my bug from the first time that code was written.