49

Across the network, users are reporting cases where tag badges are re-awarded:

They basically lose all their tag badges, and then get them all back at once:

13
  • 7
    I'd guess it's caused by this: meta.stackexchange.com/questions/359857/… -- If all the data for their tags had been deleted at some point due to that bug, their tag badges would have been lost when that process failed too. So getting the data back would also cause them all to be re-awarded.
    – animuson StaffMod
    Jan 25 at 7:38
  • 1
    *get them all back with new wrong date )= Jan 25 at 7:54
  • 11
    I've found the issue and am working on a fix for it.
    – Taryn StaffMod
    Jan 26 at 14:35
  • @Taryn just a little nitpicking: If you are working on a fix right now, isn't it status-planned then?
    – Luuklag
    Jan 26 at 16:38
  • 3
    @Luuklag considering I'm busy working on a fix, maybe nitpicking about tags isn't the right time to do it. :P
    – Taryn StaffMod
    Jan 26 at 16:39
  • Fair enough @Taryn ;)
    – Luuklag
    Jan 26 at 16:41
  • 8
    See, this is why nobody owns a status-planned tag badge :-P
    – Shog9
    Jan 26 at 18:20
  • In an ideal world, the system would delete it for being a zombie tag ...
    – Glorfindel Mod
    Jan 26 at 18:50
  • 1
    So, is the correct awarded timestamp metadata lost forever? Jan 26 at 21:04
  • 1
    @AndreSilva Probably. The previous badges were hard deleted, and I cannot imagine there's a simple way to determine when the tag badges were originally awarded before the glitch.
    – animuson StaffMod
    Jan 26 at 22:20
  • 3
    I have a screenshot of when my badge was first awarded. I'd like to get that date re-instated. It was the first gold tag badge for Excel in SuperUser.com and won by a woman. Now it looks like two men won it first. Please tell me that a bug does not take that achievement away from me.
    – teylyn
    Jan 27 at 8:51
  • 2
    @AndreSilva the awarded timestamp has been restored network wide
    – Taryn StaffMod
    Jan 28 at 19:26
  • Awsome!! Thank you very much Taryn Jan 28 at 19:46
56
+200

TL;DR; This was my fault and it's in the process of being resolved, but it is going to take a couple of days to flush out all of the issues.


Long Explanation

On Monday, January 11th I was reviewing alerts on the main SQL Servers from the weekend. I noticed that on January 10th, we had a significant spike in CPU on the main server for Stack Overflow which caused the backups to fail and lots of exceptions to be thrown.

Spiky CPU

After some investigating, I found a nightly scheduled job to update Tag Totals and assign tag badges was causing it. Initially, I hoped it was a fluke, but 2 days later we had the same sustained CPU spike - it was time to fix it (or so I thought).

The query that was running in production calculates the tag totals for every user nightly. This is fine on most of the network, but on Stack Overflow we're talking millions and millions of users and tags to calculate nightly - lots of data to process.

I took a stab at fixing the query by limiting the number of users to calculate nightly. I rewrote the query and limited it to only calculate the tag totals for Users with Posts that received a vote or a tag edit in the last week. This change significantly reduced the amount of data needed to be processed and that portion of the query performed significantly better. I tested that piece thoroughly - it was code reviewed by 3 other devs and was pushed to production on January 20, 2021. The image below is the CPU for the query once in production - no more sustained CPU spike.

Enter image description here

After this was pushed to production we started to see some exceptions in the logs when the scheduled job was running nightly. This morning, Dean Ward pointed me to some of the Meta reports about the tag/badge issues, so it was back to investigating.

It took just a few minutes to realize the issue. The job calculates the tag totals for each user, but then goes to the UserTagTotals table and deletes everything no longer needed for the user - meaning if a tag is no longer on that user's post we don't need the row any longer... well, since I had limited the totals the last 7 days it was deleting everything not in there.

Yes, I really did do this when I realized it.

Enter image description here

At this point, I have reverted the change we pushed last week and have added a hint to the "old version" to stop killing the CPU on the server. I've manually executed the job on Stack Overflow and the sites that reported the issues. The job runs nightly and will take 2 nights to clear out all of the issues.

Again, sorry for the mix-up and it should be resolved in the next few days.

12
  • "and the sites that reported the issues" I just got this on a site that wasn't reported - Code Review. Am I misunderstanding something, will this fix all the sites? Additionally will the "awarded <date> to <user>" be fixed on tag pages - /help/badges/<id>?
    – Peilonrayz
    Jan 26 at 20:52
  • 1
    @Peilonrayz I'm manually running it across all sites - initially I wasn't planning on doing that. I was going to let the scheduler handle it
    – Taryn StaffMod
    Jan 26 at 20:53
  • What is "nightly"? What time zone equivalent? Jan 26 at 21:08
  • 4
    @P.Mort.-forgotClayShirky_q 4am UTC
    – Taryn StaffMod
    Jan 26 at 21:09
  • 15
    I was the first woman to earn a gold tag badge for microsoft-excel on SuperUser.com in May 2018. That tag was lost and came back today as a new achievement with today's date. Please fix the date back to the original award date, because now it looks like two men got the badge a year before me, whereas in reality, I earned it 18 months before they did. It's rare enough that women get somewhere first in this male-dominated part of the world and I'm really proud of being the first one to win that badge. Please don't let a bug take that achievement away from me.
    – teylyn
    Jan 27 at 7:10
  • 1
    @ouflak it's possible to calculate (at least a close approximate to) historical post scores, hence tag scores, hence moments when users were eligible for the tag badges. They might also fire up an old backup, who knows ...
    – Glorfindel Mod
    Jan 27 at 9:43
  • @teylyn I hope they can get your tag date back, if not it would be worth raising a post on this meta site so it's not forgotten.
    – Pureferret
    Jan 27 at 12:13
  • 11
    @teylyn I'm looking to see if I can recreate the dates and fix it up.
    – Taryn StaffMod
    Jan 27 at 13:16
  • 1
    @teylyn how do you know they are men? Just nitpicking, congrats for the badge though.
    – 10 Rep
    Jan 28 at 0:38
  • 1
    @10Repsaysgetvaccinated LOL. We communicate on other channels. They're men.
    – teylyn
    Jan 28 at 1:48
  • 6
    @teylyn the dates for your badges have been restored
    – Taryn StaffMod
    Jan 28 at 18:31
  • Did we have another reset recently?
    – ouflak
    Jul 1 at 6:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .