Improper use of rel=canonical is hurting search

Question

When a question has many answers, the Stack Exchange software will split them onto several pages. As of mid-2009, the second and later pages have included a rel=canonical link tag pointing to the first page.

One effect of this is that search engines, such as Google, treat links to any of the pages as if they pointed to the first page. It also means that, if users search for a phrase in the question (which is shown on all pages), they won't see duplicate results.

Unfortunately, this use of rel=canonical links has another effect: it makes any answers that fall on the second or later pages impossible to search for.

For example, if you search for a distinctive phrase from this second-page answer, all you get are SO mirror sites! Meanwhile, searching for the accepted answer works fine. This may not seem like much of a loss, since answers on later pages tend to be of lower quality (although sometimes they might simply be new, and therefore not upvoted yet), but it does lose us occasional "long tail" hits from people actually searching for something found in those answers.

This (ab)use of rel=canonical might have been a useful SEO trick back in 2009, but there are better alternatives nowadays. The two solutions recommended by Google are:

1. provide a "view all" page and point the rel=canonical tag to it, or
2. use rel=next and rel=prev links instead of rel=canonical, as described here.

In fact, in this recent blog post and video, Google's Maile Ohye straight up says that the way we're currently using rel=canonical is considered "improper usage":

"While it’s fine to set rel=”canonical” from a component URL to a single view-all page, setting the canonical to the first page of a parameter-less sequence is considered improper usage. We make no promises to honor this implementation of rel=”canonical.”"

(See also the video starting at 11:42.)

Thus, I suggest that we get on with the times and fix our pagination, either by providing "view all" pages or by using the appropriate next/prev link elements.

See also:

• Original proposal to (ab)use rel=canonical in this manner (to fix duplicate search results)
• Older meta post from late 2009, pointing out the problem but providing no alternative
• Webmasters.SE post that originally brought this to my attention

---

Addendum:

In his answer below, balpha lists some problems that he suggests implementing this request would cause. Let me quote them here and respond to them point by point:

• Not marking duplicate content as such.

This issue does not apply to the solution using a canonical "view all" page, which in any case is the preferred solution suggested by Google.

It is doubtful whether (after the recent algorithm updates described in the links above) it applies to the rel=next/prev solution either: having a common header on each page of a paginated content sequence is very common, and I would be surprised if Google had neglected to deal with it appropriately. In any case, if in doubt, the easy solution would be to just ask Google (e.g. on their Webmasters forum) what they recommend.

• Adding irrelevant search results to the 99.9% case of searching for the question.

This is a non-issue for either of the proposed solutions. With a canonical "view all" pages, that page would be the only one appearing in search results. As for rel=next/prev, Google's Maile Ohye writes:

"When you implement rel="next" and rel="prev" on component pages of a series, we'll then consolidate the indexing properties from the component pages and attempt to direct users to the most relevant page/URL. This is typically the first page."

The official documentation says the same thing, namely that using rel=next/prev on a sequence of pages will generally lead to Google "consolidating their linking properties and usually sending searchers to the first page".

• Giving validity to those endless, almost forum-style question threads that we're on a mission to rid the Internet of.

I can't argue with this point, since it's primarily philosophical rather than technical in nature. I can only ask whether it's really worth making those threads effectively unsearchable.

• Spending time to work on (and risking Google punishing us for) solving a non-issue.

That's two issues; let me address them separately:

1. Yes, implementing either of these solutions would require some work for not much gain. However, the time and effort required should also be minimal (probably comparable to the time it took me to write this post, or for balpha to answer it): on one hand, we already provide rel=next/prev links in the body of the page, just not in the head where Google wants them; on the other hand, while implementing a good pagination system is a non-trivial coding exercise, providing a non-paginated alternative view should be a five minute job after that. (OK, make that fifteen minutes — it needs to be tested and deployed too.)

2. Google is not going to punish us for following their explicit recommendations. There is, however, a risk that Google may some day decide to start punishing us for doing what we currently do, which is (and always has been) against their guidelines for using rel=canonical, and which is uncomfortably close to a known black hat SEO technique.

   Admittedly, this is unlikely to happen soon or without warnings, if only because we're far from the only site currently (ab)using rel=canonical like this. (It's included in mistake #3 on this video of five common SEO mistakes.) Still, it's worth keeping in mind.

I'd also like to point out that Stack Overflow, while being the biggest SE site, may not be completely representative of other sites using this software. For example, on CodeGolf.SE beta, the very nature and purpose of the site often leads to large numbers of answers. One might well argue that this makes the Stack Exchange software poorly suited for this site (or vice versa), but — given that the site does exist and seems to be thriving — surely that's no reason to cripple the software any more than necessary.

Answer 1

• The two examples that Maile Ohye gives in that video are 1) articles (e.g. blog posts) that are spread over several pages, and 2) product pages of e-commerce sites. Neither of these examples fit our case. Each answer stands on its own, unlike the "cookie" example in the video where one page by itself doesn't make any sense.

• rel=canonical is for duplicate content – and the most important part (as far as searching goes) is indeed duplicated across pages: The question. Which, for obvious reasons, is repeated above the answers on every page. Why is it the most important part? Because the question is what people google for. They're typing a question into the search box because they need an answer, not the other way around.

• Since the voting system is designed to push the best answers to the top, the correct thing to give a user searching for "How do I solve problem X?" is the first page. If Google responds to this by sending the user to page 2 because of some obscure keyword match occuring on the second page, that means the user gets all except the 30 best answers to the question they had.

  You call this an "SEO trick"¹; given the connotation that this acronym carries, I'd rather speak of "search result optimization" here (it's not about ending up in the SERP in the first place; rather, it's about having the most relevant page in the SERP). To Google, it's all just a blurb of text. But we know that the good stuff is on page one. Page two is neither the second half of a text that should be read start-to-finish, nor is it the second half of a product inventory that we want to make sure gets sold.

• This is the most important part: If a question has more than 30 answers, something is seriously wrong. There is a good reason why this is heavily discouraged by the engine. I argue that a valid, practical, reasonably scoped question never has such a huge amount of valid answers.

  Let's look at data. Out of the six (!) open questions from 2011 that have more than thirty answers, three are of the "Please give an example of X" kind, one is borderline off-topic. The two remaining ones are iPhone development problems where the vast majority of the answers is either hardly more than a "me too", or is of the "in my particular case, I fixed this by removing the following typo in a mildly related file" kind. On one of the two questions, the downvoted answers already start on page one.

^{1 It should be noted that the original reason for adding the canonical isn't even valid anymore (that's solved by excluding non-default answer sorts via robots.txt), but I think it's still valid.}

So, what problem would your suggested change solve?

• You can google for the content of page 2-answers.

What problems would it cause?

• Not marking duplicate content as such.
• Adding irrelevant search results to the 99.9% case of searching for the question.
• Giving validity to those endless, almost forum-style question threads that we're on a mission to rid the Internet of.
• Spending time to work on (and risking Google punishing us for) solving a non-issue.