Update: we've fixed another underlying cache bug (where cache wasn't being cleared cross-server), that should eliminate the last of the craziness including the unable-to-approve scenario above.
A fix for this is being deployed across the network in the next few minutes. It's a race condition causing it...so we'll continue to monitor.
What caused this you say? We made some major performance tweaks last night to further lessen our SQL load. One thing that we were needlessly doing on every question page load was querying for suggested edits on any of the posts. This has 2 effects:
- If you made the edit, you see it with a banner that notes it's waiting for approval
- If you have an edit button, you see a (1) indicating there's an edit (which also causes the review panel to popup, rather than an editor...since there is one)
Given that even on Stack Overflow the Suggested Edit queue is capped at 200, and most of the time it's in the single digits, this meant a 99.9999% miss rate on that query. Wouldn't it be better to cache a list of Post Ids in memory that have edits, then just do a further query if there's a hit on that cache? Why yes, that sounds like a lovely idea.
What was happening was individual web servers were removing items from that list as they were dequeued (approved, rejected, deleted, etc.) but were doing so from their copy of the cache which led to all sorts of per-server and cross-server race conditions.
Instead of being that optimized we'll just sync with SQL every time there's a change. On average this is around once a minute...compare that to the query on every question show running approximately 150 times per second (this was measured well after peak hours, it actually gets run way more than that), that's still a tremendous win. Also the sync query is a 0-1ms operation with our filtered indexing, so that helps make this a no-brainer fix.
As I wrap this up, Stack Overflow is rolling through a deploy now and will finish in 90 seconds...if you still see issues after that please comment here and we'll track them down.