As part of the sidebar changes for related questions, there were 2 issues here, which we didn't understand until today.
- There was a SQL server overload (pegged at 100% for several minutes) when a query plan for the new sidebar went horribly wrong over 20 hours after deployment. A fix for this was deployed, but for a time period related questions weren't being rebuilt correctly and that includes some of those reported. This masked the second issue...
- The change itself to the sidebar has an on/off mechanism. It's currently off while we do a few more things there. The on position was fine, but the off position was buggy. A property rename was missed which was piping the new property (silently, as a
.ToString() on the collection of questions in Razor, without complaint) instead of the older StackHtmlString it should have been. This had the same symptoms as missing related questions from the database incident.
Number 2 has been fixed and I kicked off rebuilds for 1 (they should be completed by the time this is posted).
For anyone more curious about the technical bits of "on"/"off" here ("new"/"old"):
We went from caching full HTML strings for related questions (for 5 minutes - this was the "off" position) to querying it live every page load ("on"). Realistically, that cache is hit so little there's not a huge volume difference in queries. Only in hot or homepage questions is there a volume increase.
The tradeoff here is more often querying for a simplified cache, a lot less memory usage, and a generally healthier gen 2 state in our ecosystem (we're wrangling some not-compacting-as-much-as-we'd-like issues). In tuning this, we also found that:
- More columns are being brought back than we needed in the end views (for the sake of code reuse - but we can have some duplication for a page as critical as the question page to save a ton there)
- The queries could be pipelined - e.g. for collections (in Teams), linked, and related questions: we can throw those all at SQL at once for a question, eating one round-trip and read the result sets sequentially. This helps in higher latency environments and may be something we more generally do for this page overall.
- The ordering for Linked questions currently factors in views, answer count, and answer score. Due to how our Elastic indexing works w.r.t. changes to our
Post table, we moved ViewCount off to another table many ages ago - so that's a join we can eliminate if we have a roughly similar sort for popularity: e.g. Score (of the question). It was fixed here at the time, but we could get more performance and lower overhead.
Overall, these are still changes we're going to make, but some bits external to this answer are in play on the timeline. We were close on having more monitoring metrics up in our SignalFX system, so I paused the deploy here (turning the flag back "on") while we got those metrics deployed (thanks Dean!), the dashboards setup, and some decent "before" recordings before we turn this on. We had all this data before in SQL to explore...but metrics makes it even easier to compare.
Anyway, that's a short rundown of what happened here and why. I hope it's helpful!
