What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 150 Stack Exchange communities.

I was looking at robots.txt on SO and noticed that Yahoo Slurp was considered evil. I'm just wondering why?


# Yahoo bot is evil.
User-agent: Slurp
Disallow: /
share|improve this question

1 Answer 1

up vote 16 down vote accepted

It was explained in the blog a year ago, and I guess things have not really changed:

User-Agent           Requests      Bytes Served
Yahoo! Slurp/3.0       56,331     1,124,861,780
Googlebot/2.1          56,579       773,418,834
Mediapartners-Google   30,519       671,904,609

...and then, after some specific settings did not help:

In the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...

share|improve this answer
Strange, I've never seen it do that, but thanks :) +1 and marking correct, that's indeed why they banned it. – Tim Post Jun 13 '10 at 16:14
I guess that's why they called it Slurp! – Andy E Jun 13 '10 at 16:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .