# # Yahoo bot is evil. # User-agent: Slurp Disallow: /
User-Agent Requests Bytes Served Yahoo! Slurp/3.0 56,331 1,124,861,780 Googlebot/2.1 56,579 773,418,834 Mediapartners-Google 30,519 671,904,609
...and then, after some specific settings did not help:
In the meantime, since Yahoo (via Slurp!) is about 0.3% of our traffic, but insists on rudely consuming a huge chunk of our prime-time bandwidth, they’re getting IP banned and blocked. I’m a bit more sympathetic to Google, since they deliver almost 90% of our traffic, but it sure would be nice if they’d allow me to at least schedule the massive web spider storms for off-peak hours...