14

It's been pointed out that the Stack Overflow sitemap is only available to certain user agents (for instance, msnbot/2.1, Googlebot/2.1 (+http://www.google.com/bot.html), and presumably other common user agents of popular search engine bots). Jeff has already answered that this is because it's a large file and it was being abused by being downloaded too often; which is reasonable, but I think using a whitelist approach for such a restriction is short-sighted, and a blacklist (perhaps along with some other techniques) would work better.

A lazy developer who discovers this restriction on Stack Overflow (or other sites that do the same thing) is likely to just use a user-agent string from a popular search engine bot to get around the restriction, thus destroying the usefulness of filtering and logging based on user agents. A slightly less lazy developer may crate a user agent string that claims to be my agent (like msnbot/2.1), getting around the restriction but winding up in the end with crazy user agent string like browsers need in order to get around web-sites that improperly use information in user agent strings:

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-US) AppleWebKit/532.8 (KHTML, like Gecko) Chrome/4.0.302.2 Safari/532.8

An author of a new bot or search engine could also ask to be added to the whitelist; but if many sites did the same thing, it would be prohibitive to ask to be added to each and every whitelist. So, if whitelisting the sitemap becomes a common practice, then in order for new search engines or other users of bots to become competitive, they'd have to take one of the approaches above, which in turn makes it harder to effectively do any sort of whitelisting, blacklisting, or distinguishing between different bots in logs.

What are the alternatives? Well, blacklisting known bad user agents would be one. I can't imagine that new bots or other software that abuse the sitemap enough to worry about come about all that often. A blacklist would also avoid problems if one of the major search engines changed their user agent to something that didn't match the existing whitelist.

Another improvement would be to split the sitemap up using sitemap indexes. There could be a smaller sitemap index, which referred to several other sitemap files and the date they were last updated. This would reduce the load from both good and bad user agents; you would not have to send Googlebot information about pages that haven't been updated in several weeks, if you had an index which broke the sitemap down into sections that depend on how old the information is. Now, sure, a bad user agent could be pathological and try to traverse the index and all of the sitemaps within it without regard to update times or without regard to what information it actually needs, but blacklisting (based on user agent and/or IP) would probably work fine for such user agents.

Improve this question
2
  • Blacklisting causes the same impersonation problem too. If a bot is known to appear on a blacklist, then the authors can just make it claim it's Googlebot (or one of the other good bots). – ChrisF♦ Jan 27 '10 at 23:17
  • 7
    Blacklisting can cause the same problem, but is less likely to. Whitelisting means that anyone, even those that are conscientious and write good bots but happen not to be Google, have to fill their user agent with junk to get past whitelists. Blacklists mean that just abusive agents will be banned; someone who writes a bad agent through incompetence is unlikely to notice that they've been blacklisted, and someone who write a bad agent through malice will work around either a whitelist or a blacklist just the same. – Brian Campbell Jan 27 '10 at 23:30
12

+1, couldn't agree more!

Jeff's white-list will force new search engines to partly impersonate google, same reason for which all browsers impersonate mozilla (that died decades ago).

Very interesting and hilarious article on the matter: http://webaim.org/blog/user-agent-string-history/

My favorite paragraph:

And then Google built Chrome, and Chrome used Webkit, and it was like Safari, and wanted pages built for Safari, and so pretended to be Safari. And thus Chrome used WebKit, and pretended to be Safari, and WebKit pretended to be KHTML, and KHTML pretended to be Gecko, and all browsers pretended to be Mozilla, and Chrome called itself Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13, and the user agent string was a complete mess, and near useless, and everyone pretended to be everyone else, and confusion abounded.

Improve this answer
3

"What IS your fascination with my forbidden closet of mystery?"

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

Improve this answer
8
  • 2
    but but but now it's even worse.. I don't think it's fair to penalize new search engines just because they're new. – Thomas Bonini Jan 28 '10 at 7:01
  • 4
    what new search engines? I checked our logs and I see thousands (literally, thousands) of requests for sitemap.xml from regular browser user agent strings.. and none from "new search engines". – Jeff Atwood Jan 28 '10 at 7:02
  • 6
    So, would Google have been as competitive if in 1997, site operators had said "what new search engines?" and blocked all bots but AltaVista, Lycos, Yahoo, and a handful of others? Or would Apple have been able to release Safari if people hoping to block unwanted bots had blocked all user agents but IE, Mozilla, and Opera, saying "what new browsers?" Sure, new search engines, or new browsers, don't come out every day, but when they do, having access to the same websites as the existing agents can be a make-or-break sort of issue. – Brian Campbell Jan 28 '10 at 8:26
  • 4
    It's an issue of the open web. A large part of the value of the web is that it's open; open for people to write bots to parse and understand the structure, to help people search it. Open to developers who want to check out how a popular site that seems to get articles listed in Google very quickly does so. Open to people who want to view source, open to people who want to post content without getting approval from gatekeepers, open to new devices, new browsers, new ways of extracting data. If buggy software puts undue load on your server, block it; but don't block everyone else too. – Brian Campbell Jan 28 '10 at 8:31
  • 3
    And, of course, there are crawlers beyond search engines, too. Researchers might write crawlers to document how the web changes over time. Archivists might write crawlers to archive the web. They may not be using your sitemap now, but if you block them, they will never be able to. Are you really going to notice a single entry in the logs where a new crawler requests sitemap.xml and gets a 404, and based on that decides not to try requesting it any more? – Brian Campbell Jan 28 '10 at 8:42
  • 6
    @brian your claims are a little hysterical and over-dramatic; nothing's stopping a new search engine from crawling our pages at will. Also, you aren't looking at our logs every day like I am -- the number of horribly, abusively written crawlers (from large companies, even) is absolutely appalling. – Jeff Atwood Jan 28 '10 at 17:18
  • 4
    @Jeff Nothing's preventing them from crawling the site, but you provide a sitemap to the big search engines for a reason; it helps them find new content and know what content to check back on most frequently. Not providing that to other crawlers puts them at a disadvantage. I recognize the damage that abusive bots can do. I support blocking them; I just recommend a blacklist instead of a whitelist, so that new bots (and curious web devs) can have access unless they are shown to be abusive. – Brian Campbell Jan 28 '10 at 18:09
  • 5
    @Jeff This does depend on the abuse patterns you see, which I don't have access to; if you do get many new abusive agents every day, then a whitelist is easier to maintain, and I won't fault you for avoiding the time and effort constantly adding new bots to a blacklist. But if you can block the majority of abuse with a blacklist that needs updating only once every month or two, I would argue that providing more open access to your sitemap is worth that cost. This is just a suggestion, and I'm trying to make the strongest argument I can for it; I realize that you have other factors to consider. – Brian Campbell Jan 28 '10 at 18:13
2

(Unfortunately, I speak by experience)

Blacklist : a continuous work to identify and block access to clever people stealing content, running poor scripts (that impact the server performance) or trying to find a new way to regain access once blacklisted.

Whitelist: a simple and efficient way to identify and add access once to the good guys and let them make good work.

Improve this answer
4
  • 8
    Unless there are new good guys. – Thomas Bonini Feb 5 '10 at 18:45
  • I mentioned that in my comments on Jeff's answer. This does depend on the abuse pattern they are seeing. If there are so many new, abusive, user agents being created that blacklisting them would take up more than a few minutes every month, and failing to blacklist them would cost real money, then sure, use a whitelist. But whitelisting does have real costs, of blocking useful user agents without realizing it. Here's an example: Nupedia used a whitelist style approach for contributors, while Wikipedia uses a blacklist (with lots of effort to maintain). Which one is a more successful project? – Brian Campbell Feb 5 '10 at 19:25
  • New good guys can send a request to be whitelisted. But I do not know Jeff's policy for that situation. Interesting point that could be investigated. – Toto Feb 5 '10 at 22:04
  • 8
    New good guys cannot afford to do this for every site that acts this way. And in fact, how many crawlers are even going to realize that sites have whitelists; from the point of view of a crawler, it just looks like Stack Overflow is broken, as it references a non-existant sitemap from its robots.txt. This isn't about just Stack Overflow; this is about whitelisting as a general practice leading to higher cost of entry into the market. – Brian Campbell Feb 7 '10 at 22:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .