From /robots.txt on one Stack Exchange site:

# Yahoo bot is evil.
User-agent: Slurp
Disallow: /

Four years ago, someone asked what's so "evil" about Yahoo!, and the answer was that Slurp was making too many requests during peak hours. Is this still the case? And how will this affect inbound traffic from search engines to Stack Exchange sites now that Mozilla has switched default web search in Firefox from Google to Yahoo!?

  • I see the manipulation in the title you did. ;)
    – yo'
    Dec 9, 2014 at 17:37
  • 1
    1. Firefox has only about 11% browser market share. 2. Users can still change their search engine preference. 3. I don't think Stack Exchange users can be lumped in with "most users" who "do not change the default settings of their software." Dec 9, 2014 at 19:03
  • The other 'bot only says it does no evil. Dec 9, 2014 at 20:40
  • 4
    If it was really evil, it wouldn't honor robots.txt. Dec 10, 2014 at 15:41
  • @FishBelowtheIce maybe that's exactly what the comment means: "It won't follow this rule anyway but it might as well be here"
    – 3ventic
    Dec 17, 2014 at 1:04
  • duckduckgo.com is the only non-evil search engine!
    – bjb568
    Dec 17, 2014 at 19:43

1 Answer 1


Since 2009, Yahoo Search doesn’t crawl anymore. They use Bing’s index instead.

And as the robots.txt files of the Stack Exchange sites (for example, for this Meta) don’t disallow crawling for bingbot, you can find SE content by using Yahoo Search just fine:


While Yahoo! Slurp is no longer used for crawling content for Yahoo Search, it still has some other purposes, e.g., crawling of "partner sites" for inclusion in Yahoo News/Finance/Sports, and to "confirm accuracy and improve Yahoo's personalized content" for Yahoo Search.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .