6

From /robots.txt on one Stack Exchange site:

#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /

Four years ago, someone asked what's so "evil" about Yahoo!, and the answer was that Slurp was making too many requests during peak hours. Is this still the case? And how will this affect inbound traffic from search engines to Stack Exchange sites now that Mozilla has switched default web search in Firefox from Google to Yahoo!?

Improve this question
6
  • I see the manipulation in the title you did. ;)
    – yo'
    Dec 9, 2014 at 17:37
  • 1
    1. Firefox has only about 11% browser market share. 2. Users can still change their search engine preference. 3. I don't think Stack Exchange users can be lumped in with "most users" who "do not change the default settings of their software."
    – Bill the Lizard
    Dec 9, 2014 at 19:03
  • The other 'bot only says it does no evil.
    – Deer Hunter
    Dec 9, 2014 at 20:40
  • 4
    If it was really evil, it wouldn't honor robots.txt.
    – Fish Below the Ice
    Dec 10, 2014 at 15:41
  • @FishBelowtheIce maybe that's exactly what the comment means: "It won't follow this rule anyway but it might as well be here"
    – 3ventic
    Dec 17, 2014 at 1:04
  • duckduckgo.com is the only non-evil search engine!
    – bjb568
    Dec 17, 2014 at 19:43

1 Answer 1

Reset to default
8

Since 2009, Yahoo Search doesn’t crawl anymore. They use Bing’s index instead.

And as the robots.txt files of the Stack Exchange sites (for example, for this Meta) don’t disallow crawling for bingbot, you can find SE content by using Yahoo Search just fine:

http://search.yahoo.com/search?p=site%3Astackexchange.com+Is+the+Firefox+browser%27s+default+search+engine+still+evil+in+2014%3F

While Yahoo! Slurp is no longer used for crawling content for Yahoo Search, it still has some other purposes, e.g., crawling of "partner sites" for inclusion in Yahoo News/Finance/Sports, and to "confirm accuracy and improve Yahoo's personalized content" for Yahoo Search.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .