16

Likely the same problem that happened on Are the reputation notifications broken again? (April 2017) and Reputation not appearing in Rep. Notification Queue - yet it is given silently (August 2018). From what I gather from these two questions, this is a bug that happens sometimes and need an operation of sorts to be fixed. Then again, I'm not sure of the technical stuff behind it so, better to report it again just in case.

The achievements dialog does not display every rep change, though the total rep is modified as it should. Sighted on SFF and HSM. Firefox 62.0.3, Windows 10.

Wrong reputation notifications for SFF on desktop

same but on HSM

Same on mobile. Chrome 69.0.3497.100 from Google Play, phone running Android 5.1.

same bug on mobile

For what it's worth:

  • I've noticed that with the tag wiki edit one, so about five hours ago, and got confirmed when I checked the profile board;
  • I've tried turning it off and on again logging out and back again;
  • I asked Alex on SFF's chat but no repro on their part.
Improve this question
6
  • I have been having this problem as well. notifications are +5, but on my profile it is +10 for an answer today on IPS. However, I just did an experiment with another user where they upvoted a different answer of mine and it was reported as +10. Old notifications still are +5 for each vote. Seems odd.
    – BlackThorn
    Commented Oct 4, 2018 at 20:59
  • For what it's worth, all three upvotes I've had so far for this very question have shown up in the achievement dialog.
    – Jenayah
    Commented Oct 4, 2018 at 21:01
  • Also experiencing this, along with some badge icons inconsistency
    – DarkCygnus
    Commented Oct 4, 2018 at 21:11
  • Related: meta.stackexchange.com/q/315892/332286
    – DarkCygnus
    Commented Oct 4, 2018 at 21:17
  • 1
    For what it's worth, it's fixed on SFF now; issue remains on HSM.
    – Jenayah
    Commented Oct 4, 2018 at 22:16
  • WFM on 2 seperate sites.
    – Rob
    Commented Oct 5, 2018 at 0:50

2 Answers 2

Reset to default
45

I apologize for today (well I guess yesterday) being far less than awesome. As Architecture lead and main author of this set of changes, it's on me.

We deployed some massive changes that we knew were risky but deployed carefully and up until the last build of the last application they looked awesome. In a "I wish more things would break so I don't think I'm crazy" level of awesome.

Then we built stackexchange.com (the app we considered the least likely to break anything). That's when shit hit the fan. Queues shot up. Alarms went off. Queries slowed down. Our SQL Servers slammed against 20Gb network throughput limits. HTTP requests piled up. We bounced between offline and unstable for 20 minutes. Then it went downhill from there.

The aggregator (the thing that powers the top bar) was one of several pieces of infrastructure that took severe collateral damage in the process. The aggregator is very old and in need of a rewrite. Tonight a large chunk was rewritten. When we observed a severe performance degradation (only observable at production scale) after the EF Core port resulting in the aggregator being unable to keep up with incoming aggregator events, I rewrote all of the persistence in the aggregator to use Dapper instead. With help from Geoff and Jarrod this is now deployed to production and running smoother and faster.

What this doesn't solve is gaps from failures in the serialization due to order of events queued in the interim with bad IDs on the primary key fields (an ordering error in the EF Core conversion - and my fault). You may notice a few missing events in the aggregator of any type: activity, reputation, posts, etc. We will be fixing those early next week with a full backfill (something we've never attempted and have to figure out and write).

Thank you for being patient as we make some huge codebase changes on the way to .NET Core. It's never our intent to be less than awesome for you, and I hate it when we fall down. When we fail, I take it very personally. I love that the community is understanding when things go wrong - you are the most important thing to all of us here. <3

Improve this answer
5
  • 4
    "we pushed our new version, we were a little bit worried, but everything was working great! Up until it wasn't, and the world will never be the same" (ominous music) - that's the beginning of every zombie B movie ever :D
    – Jenayah
    Commented Oct 5, 2018 at 6:28
  • 13
    Man, that's the best answer a developer can ever give. Detailed, full transparency, not trying to hide the bad things, and taking responsibility. Huge kudos!!
    – Shadow Wizard
    Commented Oct 5, 2018 at 6:30
  • On a side note, any chance you plan a total rewrite on stackexchange.com so updating it won't be Mission Impossible and things like that won't happen?
    – Shadow Wizard
    Commented Oct 5, 2018 at 6:33
  • It's good for Stack Overflow to go down sometime. It helps us understand the value of it. We are so dependent on it. As long as SO doesn't start charging part (or most) of my paycheck - it's cool. It would be awesome if you write a few blog detailing your experience moving to .NET Core when it's done. Thank you very much.
    – undefined
    Commented Oct 5, 2018 at 9:39
  • meta.stackexchange.com/q/316380/309650 - is it the same problem or the other one? Anyway, it's very interesting to get more technical detailes about difficulties in migration from .NET framework to .NET Core from such a large project as SO. Or maybe there is some other place to read about it?
    – Qwertiy
    Commented Oct 5, 2018 at 21:33
12

Thanks for reporting this. We're still debugging the issue, but we've tracked it down to the Aggregator that we use to push these. When you reported this, we had a backlog of thousands of items in the queue. The queue cleared, but then it backed up again.

We have several devs digging in trying to find the issue and get it resolved. It should be resolved soon.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .