Why aren't all messages on Chat SE indexed by search engines like Google?

Question

Let's take an example. Using the Chat SE search I get 28 occurrences of the word "energizer bunny". But using Google's site specific search feature I get only 2 results. As far as I know, Chat SE is indexed by search engines. Am I missing something?

Answer 1

chat.stackexchange.com is governed by a robots.txt.

It allows to crawl the following URLs:

Allow: /transcript/
Allow: /rooms/schedule/export/
Allow: /?tab=all&sort=active&page=*
Allow: /?tab=all&sort=active&page=*&nohide=*
Allow: /

What you should notice here that it will favor the "active" tab as a crawl starting point. That means that the Google crawler will see and follow the links to chatrooms that have a decent amount of regular traffic as opposed that rather quiet ones.

For the crawler to actually find that transcript page it has to traverse a couple of links. I'm not intimate with how the Google Crawler actually does it work but if it does a broad-first approach the transcript isn't reached in a few visits. And once it has reached a transcript page it has plenty to do.

Additionally Stack Exchange offers a sitemap.xml (Sitemap: https://chat.stackexchange.com/sitemap.xml) and on the Q/A sites it is used to provide crawlers with content and direct links to about 50,000 posts. I'm not sure if that same technique is in place for chat and their transcripts (and I can't check as the sitemap.xml is IP address whitelisted). There might be a problem with that sitemap.xml based on what is in the robots.txt as it isn't updated to use the HTTPS protocol as opposed to the main sites.

Given all this I expect there to be two reasons why not everything is indexed:

• Google doesn't see too many incoming links to that domain so it only crawls it at irregular intervals and when it does will most likely see active rooms.
• The Stack Exchange robots.txt and/or Sitemap.xml have an issue that needs to be diagnosed with the Google Webmaster tools.

Answer 2

Regarding Google search, as SEO expert said here:

Google can't index the whole web, nor would they want to. They just want to index pages that have a strong likelihood of ranking so they can build the best possible search engine.

This also applies for SE chat pages. So I'm pretty sure their crawler has some logic that makes it save the page it crawls only if it contains some key words, especially when it's some internal page and not part of the main site.