Below each question and answer on Stack Exchange, there's a "link" button which generates a short permalink URL to that post for convenient linking and sharing. These URLs look like this:
(Here, the first number is the post ID and the second number is my user ID, used for tracking purposes. If I'd been logged out, that second number would not be there.)
These short URLs presumably account for a large share of our incoming links, especially since they're also used by the "share this" buttons next to each question. (Well, at least the Google+ and Twitter buttons; the Facebook button appears to be using the full URL, complete with slug.)
Disallow: /u/ Disallow: /q/ Disallow: /a/
(Yes, there's also an
Allow: / line, but don't let it mislead you: since it's the last and least specific rule, Google effectively ignores it, which anyone with a Google Webmaster Tools account can easily confirm using their robots.txt testing form.)
Why is this a bad thing?
Well, let's start by reviewing what Google actually does when it encounters a link to a disallowed URL. Contrary to popular belief, disallowing a URL does not forbid Google from indexing it; rather, it forbids Google from crawling that URL, so they will (or at least may) index it, but only based on things like link anchor text. It also means that such URLs can't pass PageRank to any other pages, since Google has no idea where that page might link (or redirect) to.
Now, if Google could crawl those permalink URLs, it would receive an HTTP 301 redirect1 to the full canonical question URL. This is (or rather, would be) a very good things, since such a redirect makes them effectively treat the link as if it had pointed directly to the canonical URL of the page.
However, since the short URLs are disallowed, Google will never crawl them and will never even see the redirect. Thus, instead of passing their PageRank to the actual question, they just end up littering Google's index. It's a bit hard to deliberately search for them, since they tend to rank poorly and have few matching keywords, but they're definitely there:
Yes, let me repeat that: our questions are receiving basically no PageRank at all from shared permalinks because of these robots.txt rules. No wonder we sometimes have SEO issues.
How can we fix it?
Simple. Just remove the three lines I quoted above from our robots.txt. We're already serving the correct 301 redirects for those URLs, so no additional work is required. Yes, it really is that simple.
Let me tentatively add the following line to the list of suggested removals:
Its original purpose was to prevent non-default answer sorts from cluttering up search results with duplicate content. However, we're already including
rel="canonical" links to the canonical question URL on those pages, so this reason is no longer valid and the line should just be removed, unless doing so would cause an unacceptable reduction in crawling speed (and/or an increase in server load).
1 Look, a link to a tag wiki! When did you last see one of those? :-)