Update (2019-7-15): A new tag engine build has been deployed and .NET Core is back in rotation.
The tag engine port had one critical bug in it: for Teams (or Channels - that's our infrastructure name), we filter which sites we're going to reload. It's simply not practical to load the delta for thousands of "sites" if they haven't changed. As an optimization here, we load in a Redis hash set which things have anything that needs a delta load (e.g. new or changed questions).
This was disabled in the code for public sites, which we always reload, but ended up enabled in the port. Since the corresponding "this stuff has changed" wasn't firing, we got a list of zero sites that needed tag engine updates.
So why did it work at all? Turns out we have a safety mechanism and assume that cache can and will disappoint everyone who touches it at some point in their life. So every 2 hours, we say to hell with that reload filter list and reload all the sites anyway. Net impact: every 2 hours new questions were appearing everywhere from that tag server instance.
We've fixed the glitch and added more monitoring and status routes - we'll be keeping an eye on it throughout the day.
We have a new version of the tag engine deployed on 1 of 3 servers (part of our move to .NET Core). It looks like it suddenly started misbehaving. I've taken that server our of rotation and we've fallen back to just the other 2, but before that I took a memory/log dump to dive into Monday.
Sorry for the trouble - it was running fine for quite some time before the delta process ate it and it's not immediately apparent from the logs what happened (or that the tag engine was aware it happened), so we'll need to dive into the memory dump and see what's hung up. It's a move from .NET Framework to .NET Core overall so there may be a fun oddity we've tripped on.
I shall update this with juicy details, or boring details, whatever we find.