Over the past couple of days, I have noticed that Stack Overflow is faster than ever. This is severely affecting my thumb twiddling time. When am I supposed to practice my thumb twiddling if I don't have to wait for the pages to load?? Please think of the thumb twiddlers when considering future performance enhancements!
We looked closely at performance over the last few weeks, particularly in the last 4 or 5 business days.
we reduced the total number of queries happening on a lot of pages (question show, question lists, user page, etc.)
we deployed a page that shows us our worst query offenders in CPU time, total duration, reads, writes, etc. and optimized the stuff at the top of the lists
we improved the way we call LINQ to SQL and moved to pre-compiled LINQ queries in certain hot code paths. This seems to have gotten much faster in .NET 4.0, we saw 8x gains over plain vanilla LINQ and 2x over SQL-queries-as-LINQ-returns.
we reduced the number of columns we return in some hot code paths, where we had effectively select * from users and posts.
we were getting sloppy about the way we handled search engines, which was causing a lot of undue load -- Google's crawler is now indexing us at, and I am not making this up, 10 requests per second which is the maximum. This is confirmed in Google Webmaster Tools. Google doesn't need all the same work done per page that, say, a user does -- and when Google hits thousands of pages in a few minutes, that can kick off a lot of background work, such as rebuilding related questions. Not expensive by itself, but when multiplied by a hundred at once.. can be quite painful.
we optimized the way we retrieve site settings and configuration, which occurs sometimes hundreds of times per page.
we had some shared lock blocking and other general inefficiencies in our Redis C# library which was causing Redis (network level cache) to be slower and more blocking that it should be. This has been rewritten to be faster and block far less.
all our database calls were not quite going through the same central hub (almost, not quite) causing too many database connections in some circumstances. This has been correctly centralized now.
lots of general profiling with .NET profiling tools to identify places where we had gotten sloppy and were doing something inefficient.