Very hot network questions found

Question

The Boardgames Stack Exchange question How would an instant or sorcery with an effect that targets work with Feather? became a Hot Network Question three times, according to its timeline:

All three events have the exact timestamp (19:14:39 UTC - shown when you click the 'toggle format' link or upon hovering) so it's probably a race condition or something.

It's not a unique case; this SEDE query finds instances on a single site, and this cross-site query lists all occurrences in the network (currently 448). The current record holders, with 44 HNQ entry events, are

• Was it ever possible to access the full EGA color space in 200-line modes? on Retrocomputing
• Using frozen light as a starship shield on Worldbuilding

Answer 1

The underlying issue here has (hopefully) been fixed. Extra post history entries have been removed.

The cause was a race condition on the hot network questions global cache:

1. We cache Hot Network Questions (HNQ) across the entire network. The cache expires every 15 minutes. Every pageload (on every site) that shows HNQ hits this cache.
2. As soon as the cache expires — at T1 —, the next request across the network will recalc the HNQ list and recache it in the global cache. This executes extremely quickly (normally in around 10-15 ms), and is completed at T4.
3. If multiple requests come through between T1 and T4, then each of these requests would calculate a new HNQ list (with the last one being completed being the one to be saved in the global cache).
   1. This is not an issue for the performance of the HNQ. Worst case scenario is that a few extra individual pageloads somewhere on the network took an extra 10ms to load.
   2. In the case when a post is returned in the list that is on HNQ for the first time (and thus does not yet have a PostHistory entry for "became hot network question"), then each of the multiple requests will attempt to add this PostHistory entry.
   3. When adding the entry (on any of the requests), we first query for all new HNQ posts (T2) on the given site that dont have the PostHistory entry, and then add the entry (T3).
4. If there are multiple requests (between T1 and T4) picking up a new HNQ post where at T2 for the request no other request had hit T3 (at which point a PostHistory entry would be recorded for the post and any subsequent requests initiated before T4 would pick them up at T2), then each one will record a PostHistory entry, leading to what was reported above (where the three PostHistory records where recorded from requests that occurred at 19:14:39.453, 19:14:39.457, 19:14:39.463).

The fix:

• Change the logic of the PostHistory insert from EF to an If not exists...insert query. This eliminates the lag between T2 and min(T3) as the cause of the duplicate entry race condition here, as the existance check occurs during the insert request.
• There is a chance that this will not completely eliminate the issue under high load, and we may need to add an UPDLOCK hint to the query as well. Decided to go incrementally with this, and will add the lock later if needed.
• The old duplicate history entries have been removed.

---

new HNQ post
concurrent requests hit cache
check hist, avoid race