most recent 30 from http://meta.stackexchange.com 2010-03-14T17:39:34Z http://meta.stackexchange.com/feeds/question/1287 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx Chris W. Rea 2009-10-11T16:33:07Z 2009-12-30T08:22:56Z

<p>I'm trying to figure out how to get Google to index my site <a href="http://www.basicallymoney.com" rel="nofollow">www.basicallymoney.com</a>. It still isn't searchable by Google &mdash; annoying.</p> <p>Anyway, I signed up for <a href="https://www.google.com/webmasters/tools" rel="nofollow">Google's Webmaster Tools</a> to check &amp; configure my site for indexing. One of the tools available there is one to verify/validate the <strong>robots.txt</strong> file. See below. It looks like Google thinks StackExchange's robots.txt is <strong>borked!</strong></p> <p>In fact, if I had to guess, there could be a Unicode/UTF-8 BOM before "User-Agent" causing Google to ignore the User-Agent line, and consequently the next "Allow: /" which looks to me to be critical for permitting indexing. Is this cause for concern? <a href="http://www.seroundtable.com/archives/017801.html" rel="nofollow">This post seems to think so</a>. Thanks.</p> <p><img src="http://img25.imageshack.us/img25/3906/robotstxtbad.png" alt="Google results from verifying robots.txt" /></p> <p><hr/> Below is what I get when I telnet to www.basicallymoney.com port 80. Note the "???" question marks ... those are likely a <a href="http://en.wikipedia.org/wiki/Byte-order%5Fmark" rel="nofollow">Unicode Byte-Order-Mark</a>.</p> <p>My request:</p> <pre><code>GET /robots.txt HTTP/1.1 Host: www.basicallymoney.com </code></pre> <p>The response:</p> <pre><code>HTTP/1.1 200 OK Content-Type: text/plain Last-Modified: Tue, 06 Oct 2009 00:28:44 GMT Accept-Ranges: bytes ETag: "04613f61b46ca1:0" Server: Microsoft-IIS/7.0 X-Powered-By: ASP.NET Date: Sun, 11 Oct 2009 16:37:50 GMT Content-Length: 775 ???User-Agent: * Allow: / Disallow: /revisions Disallow: /tags [...] </code></pre>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1296#1296 David Smith 2009-10-11T20:57:31Z 2009-10-11T20:57:31Z

<p>I'll just add that this happens on my site too.</p> <p>Those first three bytes of the file are EF BB BF which is the encoding code for UTF-8. Those characters are at the beginning of the stackoverflow.com robots.txt file too. The only difference I could tell is that the stackoverflow.com robots.txt file has some comment lines at the beginning that may be pushing the "User Agent: *" line down so the error doesn't occur on that line. Adding a couple "#" lines to the beginning of the file may fix the problem. </p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1539#1539 hFAQ 2009-10-15T03:47:20Z 2009-10-15T03:47:20Z

<p>Any update on when we can expect this to be fixed?</p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1657#1657 Anton Geraschenko 2009-10-16T04:16:37Z 2009-10-16T04:16:37Z

<p>I get the same behavior for my site, but I don't know if it's actually a problem. Looking at my webmaster tools account today, it looks like the indexing just kicked in. Here are the crawl stats<br> <img src="http://img162.imageshack.us/img162/1189/crawlstats.jpg" alt="alt text" /><br> and the crawl errors:<br> <img src="http://img162.imageshack.us/img162/7920/crawlerrors.jpg" alt="alt text" /></p> <p>It looks like the only things that were not indexed because of <code>robots.txt</code> weren't supposed to be indexed (<code>robots.txt</code> disallows indexing of anything of the form <code>/*?</code>). Is something supposed to look different?</p> <p><strong>Background:</strong> The only sitemap I submitted was my <code>/sitemap.xml</code>. This past Sunday (Oct 11), my site showed up on <code>reddit.com/r/math</code>, and yesterday morning, after my site was running beta 3 and the domain name bug was no longer a problem, I posted about it on a popular mathematics blog. I assume this increase in traffic and increase of incoming links is what caused Google to start indexing the site more seriously.</p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1661#1661 interneter 2009-10-16T05:44:40Z 2009-10-16T05:44:40Z

<p>I want to point out that if Google (or any other search engine) is unable to recognize the broken robots.txt file, the search engines would just assume the file doesn't exist and continue to crawl your site as normal (w/out restrictions). </p> <p>A broken robots text is not going to steer away search engine bots. It only becomes a problem if you mis-configured it and blocked off your entire site by accident (which is not the case here).</p> <p>But looking at the robots.txt, a bigger question that I have is why are we blocking Yahoo? I mean they are not great but certainly still one of the big three search engines out there.</p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1711#1711 Eric Koslow 2009-10-16T23:04:09Z 2009-10-18T17:09:21Z

<p>My site started to get crawled 4 days ago, but it still hasn't been indexed. Anyone having the same problem?</p> <p><strong>EDIT</strong>: After exactly 1 week from launching the site my site has been indexed by google. Tutorii is now around the 3rd result on google (after the Tutorii Twitter and Digg submission).</p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1719#1719 iamgoat 2009-10-17T04:21:42Z 2009-10-17T18:03:00Z

<p>My site has been indexed after less than two weeks!</p> <p><a href="http://www.google.com/search?q=Do+dogs+make+good+baby+sitters%3F" rel="nofollow">http://www.google.com/search?q=Do+dogs+make+good+baby+sitters%3F</a> (lol)</p> <p><strong>Edit:</strong> Now that I think about it, there are many different things that may have helped:</p> <ol> <li>Added Google Analytics to the site</li> <li>Added my <a href="http://www.google.com/addurl/" rel="nofollow">URL to Google</a></li> <li>Set my <a href="http://meta.stackexchange.com/questions/1/how-do-i-point-my-custom-domain-at-my-stackexchange-site" rel="nofollow">domain as the primary</a> so that the subdomain does a 301 redirect to prevent duplicate content issues.</li> <li>Made sure new content was added to the site regularly.</li> <li><a href="http://iamgoat.com/post.cfm/launched-petnibs-com" rel="nofollow">Blogged</a> about it</li> <li>Posted a link to my site on Twitter &amp; Facebook</li> <li>Linked to my site from StackExchange in a few places which is indexed.</li> <li>Posted a link a couple times on another forum</li> </ol>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/1745#1745 graz 2009-10-17T14:41:18Z 2009-10-17T14:41:18Z

<p>My site has also been indexed and is starting to appear in search results.</p> <ol> <li>I submitted the site using the Google webmaster site</li> <li>I used a sub-domain of an existing domain (<a href="http://ask.sqlteam.com" rel="nofollow">http://ask.sqlteam.com</a>)</li> <li>I referenced the site from a couple of blog posts</li> </ol> <p>I entered the site into google on Wednesday the 7th and had my first search click through October 12th. Google shows 43 pages from site and has most of the questions from last week.</p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/2098#2098 James 2009-10-23T15:20:33Z 2009-10-23T15:20:33Z

<p>i have the same issue with google reporting robots.txt line 1 syntax not understood.</p> <blockquote> <p>Line 1: ?User-Agent: *</p> </blockquote> <p>Site is being crawled but not indexed. Is this the reason?</p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/2330#2330 Matt Cruikshank 2009-10-29T21:58:19Z 2009-10-29T21:58:19Z

<p><a href="http://meta.stackexchange.com/questions/2325/whats-new-in-beta-4" rel="nofollow">Fixed in Beta 4.</a></p>

http://meta.stackexchange.com/questions/1287/google-still-hasnt-indexed-my-site-and-its-webmasters-tools-says-ses-robots-tx/3466#3466 Murat Yasar 2009-12-30T08:22:56Z 2009-12-30T08:22:56Z

<p>Please take a loot at the link. I faced the same problem. I found a solution from google.</p> <p><a href="http://www.yasarmurat.com/Blog/19/syntax-not-understood-error-for-robots-txt-file-in-google-webmaster-tools.aspx" rel="nofollow">http://www.yasarmurat.com/Blog/19/syntax-not-understood-error-for-robots-txt-file-in-google-webmaster-tools.aspx</a></p> <p>Regards, Murat.</p>