I was looking at robots.txt on SO and noticed that Yahoo Slurp was considered evil. I'm just wondering why?
Excerpt:
#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /
Meta Stack Exchange is a question and answer site for meta-discussion of the Stack Exchange family of Q&A websites. It only takes a minute to sign up.
Sign up to join this communityI was looking at robots.txt on SO and noticed that Yahoo Slurp was considered evil. I'm just wondering why?
Excerpt:
#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /
It was explained in the blog a year ago, and I guess things have not really changed:
User-Agent Requests Bytes Served Yahoo! Slurp/3.0 56,331 1,124,861,780 Googlebot/2.1 56,579 773,418,834 Mediapartners-Google 30,519 671,904,609
...and then, after some specific settings did not help:
In the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...