vote up 24 vote down
star
11

I'm trying to figure out how to get Google to index my site www.basicallymoney.com. It still isn't searchable by Google — annoying.

Anyway, I signed up for Google's Webmaster Tools to check & configure my site for indexing. One of the tools available there is one to verify/validate the robots.txt file. See below. It looks like Google thinks StackExchange's robots.txt is borked!

In fact, if I had to guess, there could be a Unicode/UTF-8 BOM before "User-Agent" causing Google to ignore the User-Agent line, and consequently the next "Allow: /" which looks to me to be critical for permitting indexing. Is this cause for concern? This post seems to think so. Thanks.

Google results from verifying robots.txt

Below is what I get when I telnet to www.basicallymoney.com port 80. Note the "???" question marks ... those are likely a Unicode Byte-Order-Mark.

My request:

GET /robots.txt HTTP/1.1
Host: www.basicallymoney.com

The response:

HTTP/1.1 200 OK
Content-Type: text/plain
Last-Modified: Tue, 06 Oct 2009 00:28:44 GMT
Accept-Ranges: bytes
ETag: "04613f61b46ca1:0"
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET
Date: Sun, 11 Oct 2009 16:37:50 GMT
Content-Length: 775

???User-Agent: *
Allow: /
Disallow: /revisions
Disallow: /tags
[...]
flag
I've noticed the same thing – photographr.info Oct 11 at 22:12
Do you get the same results if you run the robots.txt checker against StackOverflow's robots.txt file? – Michael Pryor Oct 15 at 13:27
2 
 @Michael: I don't know how to run the checker against a site that doesn't have my meta tag. But looking at SO's robots.txt, I notice the line "new, and hopefully fixed, as of 8/16/2009 11:41 pm PST". Did the SE fork happen before this date? – Anton Geraschenko Oct 15 at 20:07
From the discussion in the comments to my answer, it seems like this should be retagged [status-completed] or [status-resolved]. – Anton Geraschenko Oct 19 at 0:50
1 
 @Anton we've updated the robots.txt in the development build to get rid of the scary warning and we'll update the tagging once this change is deployed. – Aaron Maenpaa Oct 28 at 17:10
show 1 more comment

10 Answers

vote up 1 vote down
check

Fixed in Beta 4.

link|flag
Oh not it isn't. Line 1: ?# Syntax not understood Line 69: Sitemap: /sitemap.xml Invalid sitemap URL detected; syntax not understood – Martin Brown Nov 25 at 11:10
@Martin Brown: I believe that based on Murat's answer, the error still exists in the file due to the byte order mark (BOM) because the robots.txt file is in UTF-8 format, but should be in ansi format. However, seems like Google is just ignoring that line and going on with the rest. At least I hope, that is. Not sure why the devs can't just serve the file up in ansi format. – Scott W. Dec 30 at 18:27
vote up 5 vote down

I'll just add that this happens on my site too.

Those first three bytes of the file are EF BB BF which is the encoding code for UTF-8. Those characters are at the beginning of the stackoverflow.com robots.txt file too. The only difference I could tell is that the stackoverflow.com robots.txt file has some comment lines at the beginning that may be pushing the "User Agent: *" line down so the error doesn't occur on that line. Adding a couple "#" lines to the beginning of the file may fix the problem.

link|flag
Agreed - I think they need some blank or comment lines in there. We can't change the contents of this file ourselves, right? – Chris W. Rea Oct 11 at 21:50
I haven't tried to change it but I assume we can't. – David Smith Oct 11 at 23:08
1 
 I tried dropping it into the static content pages, but it didn't override it. So - no - we can't change this ourselves at the moment. – Corey Frang Oct 15 at 11:52
I see that some sites are still getting indexed even though this is happening for their robots.txt file too. I wonder if it is just a bug with the "Google Crawler access tool"? I tried using the Google labs "Fetch as Googlebot" tool and it fetched the robots.txt file fine and didn't see any extra characters. But it didn't try parsing the file. – David Smith Oct 15 at 14:02
vote up 3 vote down

I want to point out that if Google (or any other search engine) is unable to recognize the broken robots.txt file, the search engines would just assume the file doesn't exist and continue to crawl your site as normal (w/out restrictions).

A broken robots text is not going to steer away search engine bots. It only becomes a problem if you mis-configured it and blocked off your entire site by accident (which is not the case here).

But looking at the robots.txt, a bigger question that I have is why are we blocking Yahoo? I mean they are not great but certainly still one of the big three search engines out there.

link|flag
I assume that's inherited from SO. SO started blocking Yahoo when it didn't respect crawl rates (this was mentioned on one of the SO podcasts, but I don't remember which one). – Anton Geraschenko Oct 16 at 6:04
Must be few years ago. Yahoo does follow crawl delay settings now: help.yahoo.com/l/us/… – interneter Oct 16 at 6:13
I believe the Yahoo bot block had something to do with the Yahoo! Pipes service, not Yahoo! search (unless my memory is completely failing me). – Robert C. Cartaino Oct 16 at 23:27
Found it. "oh look, Yahoo Pipes doesn't respect robots.txt. I am SHOCKED! (ip block ban time)" -- Jeff Atwood via twitter.com/codinghorror/statuses/… – Robert C. Cartaino Oct 16 at 23:29
hmm.. still torn about this. I understand the principle behind it, but having hard time to overcome the realistic part of it (potential traffic). Here's another link. According to this page: pipes.yahoo.com/pipes/docs?doc=troubleshooting#q6/… Yahoo's Pipe Robot will respect robots.txt, I've not tested it to be true or false. – interneter Oct 17 at 9:54
show 1 more comment
vote up 3 vote down

My site has been indexed after less than two weeks!

http://www.google.com/search?q=Do+dogs+make+good+baby+sitters%3F (lol)

Edit: Now that I think about it, there are many different things that may have helped:

  1. Added Google Analytics to the site
  2. Added my URL to Google
  3. Set my domain as the primary so that the subdomain does a 301 redirect to prevent duplicate content issues.
  4. Made sure new content was added to the site regularly.
  5. Blogged about it
  6. Posted a link to my site on Twitter & Facebook
  7. Linked to my site from StackExchange in a few places which is indexed.
  8. Posted a link a couple times on another forum
link|flag
vote up 2 vote down

Any update on when we can expect this to be fixed?

link|flag
vote up 2 vote down

I get the same behavior for my site, but I don't know if it's actually a problem. Looking at my webmaster tools account today, it looks like the indexing just kicked in. Here are the crawl stats
alt text
and the crawl errors:
alt text

It looks like the only things that were not indexed because of robots.txt weren't supposed to be indexed (robots.txt disallows indexing of anything of the form /*?). Is something supposed to look different?

Background: The only sitemap I submitted was my /sitemap.xml. This past Sunday (Oct 11), my site showed up on reddit.com/r/math, and yesterday morning, after my site was running beta 3 and the domain name bug was no longer a problem, I posted about it on a popular mathematics blog. I assume this increase in traffic and increase of incoming links is what caused Google to start indexing the site more seriously.

link|flag
See interneter's response. It's probably just crawling normally. – Scott W. Oct 16 at 10:23
@Scott: If it were crawling normally, why would it have said that certain pages were restricted by robots.txt? – Anton Geraschenko Oct 16 at 14:14
@Anton: True. I have no idea. – Scott W. Oct 17 at 0:40
@Anton: you answered your question in your original post ("becase of robots.txt weren't suppose to be indexed ..."). Your robots.txt are being read correctly by Google, and the pages being blocked are just due to robots.txt specifically blocking URIs containing the /? string. I'm guessing "?" refers to page with similar content but under different sorting/filtering, and they blocked it off from Google to reduce duplication. – interneter Oct 17 at 20:07
@interneter: that's what I thought. But that suggests that there is something wrong with the premise of the original question, because it looks like Google is crawling my site properly. Am I missing something? – Anton Geraschenko Oct 18 at 2:15
show 1 more comment
vote up 1 vote down

My site has also been indexed and is starting to appear in search results.

  1. I submitted the site using the Google webmaster site
  2. I used a sub-domain of an existing domain (http://ask.sqlteam.com)
  3. I referenced the site from a couple of blog posts

I entered the site into google on Wednesday the 7th and had my first search click through October 12th. Google shows 43 pages from site and has most of the questions from last week.

link|flag
vote up 1 vote down

My site started to get crawled 4 days ago, but it still hasn't been indexed. Anyone having the same problem?

EDIT: After exactly 1 week from launching the site my site has been indexed by google. Tutorii is now around the 3rd result on google (after the Tutorii Twitter and Digg submission).

link|flag
1 
 I am having this problem. I found the following page helpful in clarifying my understanding of how Google works: google.com/support/forum/… – Chris W. Rea Oct 16 at 23:16
vote up 1 vote down

i have the same issue with google reporting robots.txt line 1 syntax not understood.

Line 1: ?User-Agent: *

Site is being crawled but not indexed. Is this the reason?

link|flag
vote up 1 vote down

Please take a loot at the link. I faced the same problem. I found a solution from google.

http://www.yasarmurat.com/Blog/19/syntax-not-understood-error-for-robots-txt-file-in-google-webmaster-tools.aspx

Regards, Murat.

link|flag
+1 The tool at asymptoticdesign.co.uk/cgi-bin/check-url.pl/… does indeed show that some weird thing "" is being served at the very beginning of robots.txt, which Google displays as a question mark and doesn't understand. It doesn't matter too much now because the first line of the file doesn't contain any information, but it would be nice to fix it. – Anton Geraschenko Dec 30 at 19:40

Your Answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.