Questions sometimes not shown on homepage

Question

While I'm writing this, this question: Can we get Complete Features Guidelines of Stack Exchange? (21 minutes old and currently at -4, see this Wayback Machine snapshot) is not shown on the Meta Stack Exchange homepage (see this Wayback Machine snapshot). Yesterday, there was a similar situation with a -5 question. A few minutes later, it reappeared (as did this question).

It happens with this very question right now (snapshot) so the question score doesn't matter. The threshold for visibility on Meta is -7/-8 instead of -3/-4 on normal sites, so the questions in the first paragraph should be visible as well. Other users have noticed the same problem on other sites. Is there anything wrong with the caching mechanism? From earlier personal experience, I believe the homepage here is cached every minute, and these questions were both 10-20 minutes old already.

Answer 1

Update (2019-7-15): A new tag engine build has been deployed and .NET Core is back in rotation.

The tag engine port had one critical bug in it: for Teams (or Channels - that's our infrastructure name), we filter which sites we're going to reload. It's simply not practical to load the delta for thousands of "sites" if they haven't changed. As an optimization here, we load in a Redis hash set which things have anything that needs a delta load (e.g. new or changed questions).

This was disabled in the code for public sites, which we always reload, but ended up enabled in the port. Since the corresponding "this stuff has changed" wasn't firing, we got a list of zero sites that needed tag engine updates.

So why did it work at all? Turns out we have a safety mechanism and assume that cache can and will disappoint everyone who touches it at some point in their life. So every 2 hours, we say to hell with that reload filter list and reload all the sites anyway. Net impact: every 2 hours new questions were appearing everywhere from that tag server instance.

We've fixed the glitch and added more monitoring and status routes - we'll be keeping an eye on it throughout the day.

---

We have a new version of the tag engine deployed on 1 of 3 servers (part of our move to .NET Core). It looks like it suddenly started misbehaving. I've taken that server our of rotation and we've fallen back to just the other 2, but before that I took a memory/log dump to dive into Monday.

Sorry for the trouble - it was running fine for quite some time before the delta process ate it and it's not immediately apparent from the logs what happened (or that the tag engine was aware it happened), so we'll need to dive into the memory dump and see what's hung up. It's a move from .NET Framework to .NET Core overall so there may be a fun oddity we've tripped on.

I shall update this with juicy details, or boring details, whatever we find.

Answer 2

I can confirm the behavior claimed in the OP happening.

Right ATM when I viewed the above question (again), and clicked at the Questions link, it wasn't listed at the homepage of Meta Stack Exchange.

I also noticed that this recently happens with other sites at the SE network.

---

I believe there must have been some recent change regarding the visibility of questions at the homepage (Newest at least).
I knew that behavior regarding the allover number of views for a question, and seen that for views less than 3 earlier. But this is new.
Especially since it seems questions appear and disappear from the homepage list arbitrarily.

---

Probably related observations (feel free to edit and add more):

• Bug when showing the newest question of a particular tag