36

I haven't been getting reputation notifications since recently.

notifications shown, with no new rep notifications

(I reloaded several times, closed and reopened my browser, and restarted my computer)

New rep is still shown in the profile page Reputation tab:

reputation tab, with new rep updates

18
  • I can reproduce: i.stack.imgur.com/XUy39.jpg
    – Luuklag
    Jul 29 at 4:36
  • Hah! I'm not getting notifications for the upvotes to this question :)
    – bobble
    Jul 29 at 4:36
  • 1
    Could have something to do with today's maintenance Jul 29 at 4:48
  • 3
    Heck, I'm not getting any... I have some followed questions and those notifications aren't coming in, either.
    – Catija StaffMod
    Jul 29 at 5:01
  • @Catija I'm getting comment notifications and chat pings, at least. And I don't have enough followed posts to check 'em.
    – bobble
    Jul 29 at 5:02
  • 2
    Yeah, I got the one for here but there's definitely others that aren't going out.
    – Catija StaffMod
    Jul 29 at 5:03
  • 1
    I'm not getting followed post notifications or reputation notifications either.
    – Ryan M
    Jul 29 at 5:38
  • same here ......
    – Mitu Raj
    Jul 29 at 6:09
  • 3
    Badge notifications do seem to be generated.
    – Glorfindel Mod
    Jul 29 at 6:42
  • On mobile I did see my updated rep bumber up top
    – Luuklag
    Jul 29 at 7:18
  • Regression of No notifications for followed posts? Jul 29 at 8:04
  • 1
    I'm only getting reputation notifications for a Teams post, nothing for the main sites.
    – Tinkeringbell Mod
    Jul 29 at 8:52
  • Same here. No notifications. But I can see the reputation change in my Profile under Reputation Tab.
    – Ram
    Jul 29 at 9:12
  • I noticed it because I got notified of a “Nice Answer” badge, but got no notification of an upvote.
    – egreg
    Jul 29 at 9:41
  • 2
    I'm getting some rep notifications now, but it hasn't backfilled fully yet. At least I assume that's what happening - I'm getting little bursts of rep notifications when I know I've repcapped overnight with this Q.
    – bobble
    Jul 29 at 12:36
31

Couple of things blew up here and it took a little while to isolate and repair the fallout, so sorry for the delay.

We had a (successful!) SQL failover in the wee hours of this morning but, for the brief period of time that the application was failing over, a number of duplicate events were written to a service that we call the "aggregator". This service is responsible for taking events from the various sites across the Stack Exchange network and aggregating them to denormalized tables so that we can get performant queries for things that cross the whole network - things like global inbox messages, rep, achievements, etc.

Usually this would be a non-issue - we have de-duplication logic in those code paths and it has functioned just fine for years. However, there were a specific set of events that had, clearly, never been de-duped - they're relatively rare (happen once per user per day). De-duping takes the form of sorting and eliminating the duplicates using equality checks - unfortunately the code for checking equality in one of these events looked like this:

public bool Equals(object obj)
{
    if (!EqualityBase(obj, out VisitEvent other)) return false;

    return other.Equals(this) && other.UserHistoryId == UserHistoryId;
}

That call to other.Equals(this) was very rarely hit, but it clearly recurses infinitely, leading to a StackOverflowException. That exception kills the process and because of the way the aggregator service runs, it meant that whichever server picked up the work would then die off when it tried to pick up the work.

We have some poison message handling around this but it didn't kick in effectively here and we have a large backlog of events to get through now the bug is fixed. It's getting through them now though and we should be all set in the next hour or so.

UPDATE Annnnnd it’s done. Thanks for your patience folks!

8
  • 23
    So Stack Overflow's rep problem was due to a Stack Overflow? (Someone had to say it :D)
    – bobble
    Jul 29 at 12:41
  • 9
    @bobble Frankly I'm only glad the site was up enough for me to find the right procdump incantation to capture a dump of a StackOverflowException :)
    – Dean Ward
    Jul 29 at 12:42
  • 1
    In TeX it would be \def\recurse{\recurse}\recurse ;-) Good job!
    – egreg
    Jul 29 at 13:15
  • Reminds me of this... Jul 29 at 13:32
  • @egreg Hate to be a nitpicker, but this infinite loop will not lead to a stack overflow, as TeX’s macro expansion routines handle tail recursion gracefully. You’d need something like \def\recurse{\recurse x}\recurse. Jul 29 at 13:37
  • @EmilJeřábek Indeed, it does even worse than causing an overflow! ;-)
    – egreg
    Jul 29 at 14:01
  • Hey Dean, its still not 100% up to speed for me.
    – Luuklag
    Jul 29 at 15:57
  • Just going to unaccept this for now since my notifications still aren't all there :) hope you understand
    – bobble
    Jul 29 at 17:24
3

As Dean estimated things would be back to normal by now, I have to say things are not lining up for me yet.

See this screenshot for example:

enter image description here

My achievements box shows 130 rep for the day, while my profile shows that I'm rep-capped at 200.

4
  • Same here - my inbox only shows 172 meta rep for today.
    – bobble
    Jul 29 at 17:23
  • 4
    It’s quite possible that those were dropped on the floor; I’m out for a few days now but somebody else on the team will continue digging here to see what’s up
    – Dean Ward
    Jul 29 at 17:30
  • Enjoy your time @DeanWard.l!
    – Luuklag
    Jul 29 at 17:37
  • 2
    Today it appears to work fine. However there is still rep dropped from yesterday in the top.
    – Luuklag
    Jul 30 at 5:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .