It's been pointed out that the Stack Overflow sitemap is only available to certain user agents (for instance, msnbot/2.1
, Googlebot/2.1 (+http://www.google.com/bot.html)
, and presumably other common user agents of popular search engine bots). Jeff has already answered that this is because it's a large file and it was being abused by being downloaded too often; which is reasonable, but I think using a whitelist approach for such a restriction is short-sighted, and a blacklist (perhaps along with some other techniques) would work better.
A lazy developer who discovers this restriction on Stack Overflow (or other sites that do the same thing) is likely to just use a user-agent string from a popular search engine bot to get around the restriction, thus destroying the usefulness of filtering and logging based on user agents. A slightly less lazy developer may crate a user agent string that claims to be my agent (like msnbot/2.1)
, getting around the restriction but winding up in the end with crazy user agent string like browsers need in order to get around web-sites that improperly use information in user agent strings:
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-US) AppleWebKit/532.8 (KHTML, like Gecko) Chrome/4.0.302.2 Safari/532.8
An author of a new bot or search engine could also ask to be added to the whitelist; but if many sites did the same thing, it would be prohibitive to ask to be added to each and every whitelist. So, if whitelisting the sitemap becomes a common practice, then in order for new search engines or other users of bots to become competitive, they'd have to take one of the approaches above, which in turn makes it harder to effectively do any sort of whitelisting, blacklisting, or distinguishing between different bots in logs.
What are the alternatives? Well, blacklisting known bad user agents would be one. I can't imagine that new bots or other software that abuse the sitemap enough to worry about come about all that often. A blacklist would also avoid problems if one of the major search engines changed their user agent to something that didn't match the existing whitelist.
Another improvement would be to split the sitemap up using sitemap indexes. There could be a smaller sitemap index, which referred to several other sitemap files and the date they were last updated. This would reduce the load from both good and bad user agents; you would not have to send Googlebot information about pages that haven't been updated in several weeks, if you had an index which broke the sitemap down into sections that depend on how old the information is. Now, sure, a bad user agent could be pathological and try to traverse the index and all of the sitemaps within it without regard to update times or without regard to what information it actually needs, but blacklisting (based on user agent and/or IP) would probably work fine for such user agents.