What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

Below each question and answer on Stack Exchange, there's a "link" button which generates a short permalink URL to that post for convenient linking and sharing. These URLs look like this:

http://stackoverflow.com/q/7729898/411022
http://stackoverflow.com/a/7730040/411022

(Here, the first number is the post ID and the second number is my user ID, used for tracking purposes. If I'd been logged out, that second number would not be there.)

These short URLs presumably account for a large share of our incoming links, especially since they're also used by the "share this" buttons next to each question. (Well, at least the Google+ and Twitter buttons; the Facebook button appears to be using the full URL, complete with slug.)

However, thanks to balpha, I just found out that our robots.txt file is blocking Google and other search engines from crawling those URLs, since it contains these lines:

Disallow: /u/
Disallow: /q/
Disallow: /a/

(Yes, there's also an Allow: / line, but don't let it mislead you: since it's the last and least specific rule, Google effectively ignores it, which anyone with a Google Webmaster Tools account can easily confirm using their robots.txt testing form.)

Why is this a bad thing?

Well, let's start by reviewing what Google actually does when it encounters a link to a disallowed URL. Contrary to popular belief, disallowing a URL does not forbid Google from indexing it; rather, it forbids Google from crawling that URL, so they will (or at least may) index it, but only based on things like link anchor text. It also means that such URLs can't pass PageRank to any other pages, since Google has no idea where that page might link (or redirect) to.

Now, if Google could crawl those permalink URLs, it would receive an HTTP 301 redirect1 to the full canonical question URL. This is (or rather, would be) a very good things, since such a redirect makes them effectively treat the link as if it had pointed directly to the canonical URL of the page.

However, since the short URLs are disallowed, Google will never crawl them and will never even see the redirect. Thus, instead of passing their PageRank to the actual question, they just end up littering Google's index. It's a bit hard to deliberately search for them, since they tend to rank poorly and have few matching keywords, but they're definitely there:

Yes, let me repeat that: our questions are receiving basically no PageRank at all from shared permalinks because of these robots.txt rules. No wonder we sometimes have SEO issues.

How can we fix it?

Simple. Just remove the three lines I quoted above from our robots.txt. We're already serving the correct 301 redirects for those URLs, so no additional work is required. Yes, it really is that simple.

Addendum:

Let me tentatively add the following line to the list of suggested removals:

Disallow: /questions/*answertab=*

Its original purpose was to prevent non-default answer sorts from cluttering up search results with duplicate content. However, we're already including rel="canonical" links to the canonical question URL on those pages, so this reason is no longer valid and the line should just be removed, unless doing so would cause an unacceptable reduction in crawling speed (and/or an increase in server load).

1 Look, a link to a tag wiki! When did you last see one of those? :-)

share|improve this question
    
I did a Google+ share of an answer recently, and the /a/ link that was put in the Google+ post had no title or summary. I wonder if this is the reason for that; if so, it's another reason to unblock them. I was rather annoyed at having the choice of using the no-tracking link or having an ugly post... –  Kevin Reid May 1 '12 at 17:36

2 Answers 2

up vote 7 down vote accepted

We have removed /q/ and /a/ from robots.txt. We shall see if this makes any impact on search and report back. SEO is somewhat of a black magic art at times.

share|improve this answer

"Yes, it really is that simple."

Not that we won't do this (we very likely will for /q/ and /a/), but you're making some assumptions here. For example those links are used for referral tracking, if we let search engines hit them, we have to exclude search engine crawls from generating referral counts and granting badges. Now we happen to be doing this already, but never assume how simple something is.

The /u/ is a shortcut route I added for quick access and quite literally because I was typing /users/ 100+ times a day, nothing more...there should be no permalinks to this, it's not intended for linking, it needn't be crawled.

"Let me tentatively add the following line to the list of suggested removals:

Disallow: /questions/*answertab=*

Its original purpose was to prevent non-default answer sorts from cluttering up search results with duplicate content."

While true, that wasn't its only purpose. We know how Google crawls us (we log every single request), and we saw a lot of needless duplicate crawls on the page tabs.

Are they canonical-ed? Sure. Does Google actually need to crawl the answers tabs to see the same content 3 ways to see they're canonical? No...it's a waste of bandwidth, and continues to be one. Their crawl rate is throttled at all times (even though we don't set one, Google self-throttles), those requests are always better spent indexing something useful than running into a canonical.


Since this comes up often, here's quick stats from last 4/27/2012 (last full weekday) for GoogleBot:

  • 2,279,580 Requests made (16.6% of all page views, 4.24% of all requests)
  • 12,528,008,337 Bytes sent (4.5% of all HTTP requests)

...so you can imagine, how efficiently they crawl us is a concern we keep in mind.

share|improve this answer
    
Good points, both of them. The second one is why I included the "unless" in the addendum; I guess there's a need to balance crawling efficiency with PageRank conservation. –  Ilmari Karonen Apr 30 '12 at 14:37
    
Nice. As for bandwith: Jeff wrote in June 2009 that Google used 737 MB per day (and Yahoo! Slurp as much as 1 GB). Is Google still crawling that much? –  Arjan Apr 30 '12 at 14:38
    
Regarding /u/, if there are no links to those URLs then they won't be crawled anyway. If there are a few links to them floating around, surely we want those links to be correctly followed by search engines? –  Ilmari Karonen Apr 30 '12 at 14:45
    
@IlmariKaronen - not really, there are plenty of links to users with any content that would get any traffic anyway. The few extra that shouldn't be laying around anyway are really spare change and insignificant. –  Nick Craver Apr 30 '12 at 14:46
3  
@Arjan - it's quite a bit more than that now since we've grown...I put some quick stats from our logs at the bottom of the answer for you. –  Nick Craver Apr 30 '12 at 14:46
11  
Think of it like Google giving Stack Overflow an 11.5 GB hug every day. - David –  jadarnel27 Apr 30 '12 at 15:12
    
Does Google ever really leave the site? New questions turn up in searches typically in a matter of minutes and it's obviously not getting them from feeds. –  Tim Post May 1 '12 at 6:21
    
@TimPost: We do ping Google every time a question is added (pubsubhubbub), though I don't know if that's in any way tied into GoogleBot, it's only used to update their feed distributions AFAIK. –  Nick Craver May 2 '12 at 0:19
    
Nick, you might be able to decrease the number of Google page hits by about 60,000? ;-) (source) –  Arjan May 2 '12 at 6:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .