6

From /robots.txt on one Stack Exchange site:

#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /

Four years ago, someone asked what's so "evil" about Yahoo!, and the answer was that Slurp was making too many requests during peak hours. Is this still the case? And how will this affect inbound traffic from search engines to Stack Exchange sites now that Mozilla has switched default web search in Firefox from Google to Yahoo!?

  • I see the manipulation in the title you did. ;) – yo' Dec 9 '14 at 17:37
  • 1
    1. Firefox has only about 11% browser market share. 2. Users can still change their search engine preference. 3. I don't think Stack Exchange users can be lumped in with "most users" who "do not change the default settings of their software." – Bill the Lizard Dec 9 '14 at 19:03
  • The other 'bot only says it does no evil. – Deer Hunter Dec 9 '14 at 20:40
  • 4
    If it was really evil, it wouldn't honor robots.txt. – Fish Below the Ice Dec 10 '14 at 15:41
  • @FishBelowtheIce maybe that's exactly what the comment means: "It won't follow this rule anyway but it might as well be here" – 3ventic Dec 17 '14 at 1:04
  • duckduckgo.com is the only non-evil search engine! – bjb568 Dec 17 '14 at 19:43
8

Since 2009, Yahoo Search doesn’t crawl anymore. They use Bing’s index instead.

And as the robots.txt files of the Stack Exchange sites (for example, for this Meta) don’t disallow crawling for bingbot, you can find SE content by using Yahoo Search just fine:

http://search.yahoo.com/search?p=site%3Astackexchange.com+Is+the+Firefox+browser%27s+default+search+engine+still+evil+in+2014%3F

While Yahoo! Slurp is no longer used for crawling content for Yahoo Search, it still has some other purposes, e.g., crawling of "partner sites" for inclusion in Yahoo News/Finance/Sports, and to "confirm accuracy and improve Yahoo's personalized content" for Yahoo Search.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .