8

I've asked this question before on Why is the site SharePoint Stack Exchange not to be found using Bing Search?, but was referred to this site:

For some reason I'm unable to find the homepage of SharePoint Stack Exchange using the keywords "SharePoint Stack Exchange".

I'm able to find the Stack Overflow and Stack Exchange sites where SharePoint tags are used. But I cannot seem to find the homepage http://sharepoint.stackexchange.com. Is by any chance the Bing crawler excluded in robots.txt?

The reason I'm posting this is because it's possible new visitors are unable to discover the site if it is not to be found using all search engines.

It would be nice if the site could be found using all (big) search engines.

EDIT:

In the other topic it was found that it is perhaps due to the robots.txt crawler settings for the Yahoo Crawler:

When answering what crawlers do and doesn't do, one is out on thin ice. But you have a point that the start page doesn't show up in the 154 results of your search string on Bing.com.

Facts are that the robots.txt file disallow all crawling from Yahoo! With the comment saying

#
# Yahoo bot is evil.
#
User-agent: Slurp
Disallow: /

Now, we know that Bing and Yahoo have a ten year agreement tocooperate in search applications, than there might be a possibility that they share crawl results with each other. If so, and the Yahoo crawler Slurp visits SP.SE it actually follows the robots.txt and shares its non existing result with Bing.

But this is just a qualified guess, where we need to talk to SEO-guys about it, or maybe follow up your question on Meta.SE!

Good catch though!

source: https://sharepoint.meta.stackexchange.com/a/1159/9256

  • 3
    Because Bing is a horrible and terrible search engine. Really. – Shadow Wizard Jan 27 '15 at 13:54
  • 1
    The problem is probably to be found in the robots.txt where the yahoo crawler is denied access (found by Benny Skogberg). I'd like to have someone who knows about the site setup have a look. That being said, opinions and preferences vary about search engines. – TempaC Jan 27 '15 at 14:07
  • 1
    True that, sorry for being harsh but in the few times I did try Bing it was.... well, horrible and terrible. Anyway, I do remember Stack Exchange blocking Yahoo Slurp for flooding them with thousands of useless requests, but not Bing. If Bing is identifying itself as Yahoo bot it's really Microsoft's problem. (If You Can't Buy Them - Pretend To Be Them? ;-)) – Shadow Wizard Jan 27 '15 at 14:09
  • 1
    You should add the information about the crawler into the post. – Mad Scientist Jan 27 '15 at 14:12
  • 1
    added the crawler info into the post. – TempaC Jan 27 '15 at 14:27
  • Needs more certain video content before Bing will index it – random Jan 27 '15 at 14:48
  • 7
    The fact that Bing indexes every other Stack Exchange site in existence proves it has nothing to do with Yahoo or the block of its user agent. Bing is run by Microsoft, which owns Sharepoint, and I have that odd feeling they are being greedy over the "Sharepoint" keyword and ignoring the Stack Exchange site to keep it out of the top results, where they'd rather the traffic be directed to them. – animuson Jan 27 '15 at 14:54
  • 2
    Ever see someone you don't really know being attacked and felt the need to step in? Bing isn't a horrible/terrible search engine, and the idea that MS is trying to get traffic directed to them is an insane conspiracy theory. Stop it. – Won't Jan 27 '15 at 17:16
  • @animuson The thought has crossed my mind too, and the only thing we can do is to continue with the search engine we know that works for us in the business of SharePoint. – Benny Skogberg Jan 27 '15 at 18:03
5

That's a bug in Bing, not Stack Exchange. Don't really care why, and really not surprised, but you can simply use a real search engine, e.g. Google.

Now for some technical details. Stack Exchange is blocking some web crawlers, according to the recent https://sharepoint.stackexchange.com/robots.txt file those are:

  • Mediapartners-Google (full block)
  • Slurp (full block)
  • Yahoo Pipes 1.0 (full block)
  • Googlebot-Image (partial block e.g. to prevent flairs from being crawled)

However, Bing is using none of the above names according to this official page it's using the following user agent strings:

  • Bingbot
  • Adidxbot
  • MSNBot
  • BingPreview

So Bing should happily crawl Stack Exchange sites, including the home page of any site. Failing to do so is really a problem of the crawler/search engine.

  • 3
    Not sure why you're downvoting Shadow Wizard, he's right. Bing is just off their rocker. I can confirm we're not blocking them. – Haney Jan 27 '15 at 22:08
  • @Haney thanks for the vote of confidence! Guess the comments on the question can explain some of the disagreement. – Shadow Wizard Jan 27 '15 at 22:46
1

And suddenly (as of today), without explanation the site is to be found using Bing.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .