What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

I was looking at robots.txt on SO and noticed that Yahoo Slurp was considered evil. I'm just wondering why?

Excerpt:

#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /
share|improve this question
add comment

1 Answer

up vote 15 down vote accepted

It was explained in the blog a year ago, and I guess things have not really changed:

User-Agent           Requests      Bytes Served
Yahoo! Slurp/3.0       56,331     1,124,861,780
Googlebot/2.1          56,579       773,418,834
Mediapartners-Google   30,519       671,904,609

...and then, after some specific settings did not help:

In the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...

share|improve this answer
    
Strange, I've never seen it do that, but thanks :) +1 and marking correct, that's indeed why they banned it. –  Tim Post Jun 13 '10 at 16:14
3  
I guess that's why they called it Slurp! –  Andy E Jun 13 '10 at 16:24
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .