14

I was looking at robots.txt on SO and noticed that Yahoo Slurp was considered evil. I'm just wondering why?

Excerpt:

#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /
Improve this question

1 Answer 1

Reset to default
17

It was explained in the blog a year ago, and I guess things have not really changed:

User-Agent           Requests      Bytes Served
Yahoo! Slurp/3.0       56,331     1,124,861,780
Googlebot/2.1          56,579       773,418,834
Mediapartners-Google   30,519       671,904,609

...and then, after some specific settings did not help:

In the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...

Improve this answer
3
  • Strange, I've never seen it do that, but thanks :) +1 and marking correct, that's indeed why they banned it.
    – Tim Post
    Jun 13, 2010 at 16:14
  • 4
    I guess that's why they called it Slurp!
    – Andy E
    Jun 13, 2010 at 16:24
  • Does SO, which is used from around the web, really got off-peak hours? just curious. Also statistics might be evil, because if used by referrals, Yahoo\Verizon may hide what traffic they bring. But I agree, Verizon is evil, so Yahoo might be too.
    – Swift
    May 4, 2018 at 7:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .