Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 155 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

I was looking at robots.txt on SO and noticed that Yahoo Slurp was considered evil. I'm just wondering why?

Excerpt:

#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /
share|improve this question
up vote 16 down vote accepted

It was explained in the blog a year ago, and I guess things have not really changed:

User-Agent           Requests      Bytes Served
Yahoo! Slurp/3.0       56,331     1,124,861,780
Googlebot/2.1          56,579       773,418,834
Mediapartners-Google   30,519       671,904,609

...and then, after some specific settings did not help:

In the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...

share|improve this answer
    
Strange, I've never seen it do that, but thanks :) +1 and marking correct, that's indeed why they banned it. – Tim Post Jun 13 '10 at 16:14
3  
I guess that's why they called it Slurp! – Andy E Jun 13 '10 at 16:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .