The underlying issue here has (hopefully) been fixed. Extra post history entries will be cleaned in the coming days.
The cause was a race condition on the hot network questions global cache:
- We cache Hot Network Questions (HNQ) across the entire network. The cache expires every 15 minutes. Every pageload (on every site) that shows HNQ hits this cache.
- As soon as the cache expires — at
T1 —, the next request across the network will recalc the HNQ list and recache it in the global cache. This executes extremely quickly (normally in around 10-15 ms), and is completed at
- If multiple requests come through between
T4, then each of these requests would calculate a new HNQ list (with the last one being completed being the one to be saved in the global cache).
- This is not an issue for the performance of the HNQ. Worst case scenario is that a few extra individual pageloads somewhere on the network took an extra 10ms to load.
- In the case when a post is returned in the list that is on HNQ for the first time (and thus does not yet have a
PostHistory entry for "became hot network question"), then each of the multiple requests will attempt to add this
- When adding the entry (on any of the requests), we first query for all new HNQ posts (
T2) on the given site that dont have the
PostHistory entry, and then add the entry (
- If there are multiple requests (between
T4) picking up a new HNQ post where at
T2 for the request no other request had hit
T3 (at which point a
PostHistory entry would be recorded for the post and any subsequent requests initiated before
T4 would pick them up at
T2), then each one will record a
PostHistory entry, leading to what was reported above (where the three
PostHistory records where recorded from requests that occurred at
- Change the logic of the
PostHistory insert from EF to an
If not exists...insert query. This eliminates the lag between
min(T3) as the cause of the duplicate entry race condition here, as the existance check occurs during the insert request.
- There is a chance that this will not completely eliminate the issue under high load, and we may need to add an
UPDLOCK hint to the query as well. Decided to go incrementally with this, and will add the lock later if needed.
- We'll be cleaning up the old duplicate history entries shortly, will update here when this is complete.
new HNQ post
concurrent requests hit cache
check hist, avoid race