I'm curious if SO uses any compression algorithm to reduce storage space of posts?
Or is this an unnecessary overhead since storage is cheap?
Meta Stack Exchange is a question and answer site for meta-discussion of the Stack Exchange family of Q&A websites. It only takes a minute to sign up.Sign up to join this community
This answer is a little trickier than Robert implies in his answer.
SQL Server 2008 introduced data compression as a feature in Enterprise Edition. Pages are compressed before they're written to disk automatically, but only for the objects you choose - not just at the table level, but at the index level. I might have a list of users that I keep compressed (because they have huge about-me profiles that compress very well with low CPU overhead) and a list of indexes that I keep uncompressed (because I hit those constantly).
At first glance, it would seem like compression would slow things down, but that's not usually the case with database compression. It turns out that in most cases, we're waiting on disk anyway, and if we can write less to disk by doing some quick CPU calculations, the overall query time will decrease. CPU load goes up, but query time goes down, and SQL Server spends less time switching between tasks waiting for stored data. SQL Server's compression works very well for OLAP style data - huge volumes of data that aren't changed very frequently.
However, one of the areas where it doesn't perform as well is OLTP-style data where pages are constantly changing, like StackOverflow's setup.
If we partitioned off the older data (questions that are stale), we could get a performance boost, but partitioning has its own drawbacks, and I've written about that: