What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

When a question has many answers, the Stack Exchange software will split them onto several pages. As of mid-2009, the second and later pages have included a rel=canonical link tag pointing to the first page.

One effect of this is that search engines, such as Google, treat links to any of the pages as if they pointed to the first page. It also means that, if users search for a phrase in the question (which is shown on all pages), they won't see duplicate results.

Unfortunately, this use of rel=canonical links has another effect: it makes any answers that fall on the second or later pages impossible to search for.

For example, if you search for a distinctive phrase from this second-page answer, all you get are SO mirror sites! Meanwhile, searching for the accepted answer works fine. This may not seem like much of a loss, since answers on later pages tend to be of lower quality (although sometimes they might simply be new, and therefore not upvoted yet), but it does lose us occasional "long tail" hits from people actually searching for something found in those answers.

This (ab)use of rel=canonical might have been a useful SEO trick back in 2009, but there are better alternatives nowadays. The two solutions recommended by Google are:

  1. provide a "view all" page and point the rel=canonical tag to it, or
  2. use rel=next and rel=prev links instead of rel=canonical, as described here.

In fact, in this recent blog post and video, Google's Maile Ohye straight up says that the way we're currently using rel=canonical is considered "improper usage":

"While it’s fine to set rel=”canonical” from a component URL to a single view-all page, setting the canonical to the first page of a parameter-less sequence is considered improper usage. We make no promises to honor this implementation of rel=”canonical.”"

(See also the video starting at 11:42.)

Thus, I suggest that we get on with the times and fix our pagination, either by providing "view all" pages or by using the appropriate next/prev link elements.

See also:


Addendum:

In his answer below, balpha lists some problems that he suggests implementing this request would cause. Let me quote them here and respond to them point by point:

  • Not marking duplicate content as such.

This issue does not apply to the solution using a canonical "view all" page, which in any case is the preferred solution suggested by Google.

It is doubtful whether (after the recent algorithm updates described in the links above) it applies to the rel=next/prev solution either: having a common header on each page of a paginated content sequence is very common, and I would be surprised if Google had neglected to deal with it appropriately. In any case, if in doubt, the easy solution would be to just ask Google (e.g. on their Webmasters forum) what they recommend.

  • Adding irrelevant search results to the 99.9% case of searching for the question.

This is a non-issue for either of the proposed solutions. With a canonical "view all" pages, that page would be the only one appearing in search results. As for rel=next/prev, Google's Maile Ohye writes:

"When you implement rel="next" and rel="prev" on component pages of a series, we'll then consolidate the indexing properties from the component pages and attempt to direct users to the most relevant page/URL. This is typically the first page."

The official documentation says the same thing, namely that using rel=next/prev on a sequence of pages will generally lead to Google "consolidating their linking properties and usually sending searchers to the first page".

  • Giving validity to those endless, almost forum-style question threads that we're on a mission to rid the Internet of.

I can't argue with this point, since it's primarily philosophical rather than technical in nature. I can only ask whether it's really worth making those threads effectively unsearchable.

  • Spending time to work on (and risking Google punishing us for) solving a non-issue.

That's two issues; let me address them separately:

  1. Yes, implementing either of these solutions would require some work for not much gain. However, the time and effort required should also be minimal (probably comparable to the time it took me to write this post, or for balpha to answer it): on one hand, we already provide rel=next/prev links in the body of the page, just not in the head where Google wants them; on the other hand, while implementing a good pagination system is a non-trivial coding exercise, providing a non-paginated alternative view should be a five minute job after that. (OK, make that fifteen minutes — it needs to be tested and deployed too.)

  2. Google is not going to punish us for following their explicit recommendations. There is, however, a risk that Google may some day decide to start punishing us for doing what we currently do, which is (and always has been) against their guidelines for using rel=canonical, and which is uncomfortably close to a known black hat SEO technique.

    Admittedly, this is unlikely to happen soon or without warnings, if only because we're far from the only site currently (ab)using rel=canonical like this. (It's included in mistake #3 on this video of five common SEO mistakes.) Still, it's worth keeping in mind.

I'd also like to point out that Stack Overflow, while being the biggest SE site, may not be completely representative of other sites using this software. For example, on CodeGolf.SE beta, the very nature and purpose of the site often leads to large numbers of answers. One might well argue that this makes the Stack Exchange software poorly suited for this site (or vice versa), but — given that the site does exist and seems to be thriving — surely that's no reason to cripple the software any more than necessary.

share|improve this question
3  
+1, but consider changing the title to something like "Stack Exchange's improper rel values are hurting search". –  Awesome Poodles Apr 30 '12 at 0:54
    
@Brock: I was aiming for "neutral and descriptive" with the current title, but I'm happy to change if it others think it sucks too. –  Ilmari Karonen Apr 30 '12 at 1:13
    
16 views and only 3 votes; this question is better than that. IMO, the title seems esoteric and dry, but I didn't make the edit myself because it is a subjective call. –  Awesome Poodles Apr 30 '12 at 1:20
    
Well, it's barely half an hour old. –  Ilmari Karonen Apr 30 '12 at 1:24
1  
@BrockAdams It is the end of the weekend, I expect it will "blow-up" on Monday. –  Hogan Apr 30 '12 at 3:32
    
It's probably an argument for people to curb the piling on or to cull the mulesing cruft –  random Apr 30 '12 at 3:38
2  
Not an argument against this request, but keep in mind that this problem is very limited in scope. There seems to only be 620 or so questions on Stack Overflow that have multiple pages of answers, nearly half of which are closed, and only a fourth of which have positively-voted answers on the second page. So, at least it's unlikely to have negatively impacted the general search experience. –  Tim Stone Apr 30 '12 at 4:05
4  
@Hogan, he made the change already and it took off. Went from 4 votes to 21+ Sadly, I cannot seem to get a 20% commission. (^_^) –  Awesome Poodles Apr 30 '12 at 4:51

1 Answer 1

  • The two examples that Maile Ohye gives in that video are 1) articles (e.g. blog posts) that are spread over several pages, and 2) product pages of e-commerce sites. Neither of these examples fit our case. Each answer stands on its own, unlike the "cookie" example in the video where one page by itself doesn't make any sense.

  • rel=canonical is for duplicate content – and the most important part (as far as searching goes) is indeed duplicated across pages: The question. Which, for obvious reasons, is repeated above the answers on every page. Why is it the most important part? Because the question is what people google for. They're typing a question into the search box because they need an answer, not the other way around.

  • Since the voting system is designed to push the best answers to the top, the correct thing to give a user searching for "How do I solve problem X?" is the first page. If Google responds to this by sending the user to page 2 because of some obscure keyword match occuring on the second page, that means the user gets all except the 30 best answers to the question they had.

    You call this an "SEO trick"1; given the connotation that this acronym carries, I'd rather speak of "search result optimization" here (it's not about ending up in the SERP in the first place; rather, it's about having the most relevant page in the SERP). To Google, it's all just a blurb of text. But we know that the good stuff is on page one. Page two is neither the second half of a text that should be read start-to-finish, nor is it the second half of a product inventory that we want to make sure gets sold.

  • This is the most important part: If a question has more than 30 answers, something is seriously wrong. There is a good reason why this is heavily discouraged by the engine. I argue that a valid, practical, reasonably scoped question never has such a huge amount of valid answers.

    Let's look at data. Out of the six (!) open questions from 2011 that have more than thirty answers, three are of the "Please give an example of X" kind, one is borderline off-topic. The two remaining ones are iPhone development problems where the vast majority of the answers is either hardly more than a "me too", or is of the "in my particular case, I fixed this by removing the following typo in a mildly related file" kind. On one of the two questions, the downvoted answers already start on page one.

1 It should be noted that the original reason for adding the canonical isn't even valid anymore (that's solved by excluding non-default answer sorts via robots.txt), but I think it's still valid.

So, what problem would your suggested change solve?

  • You can google for the content of page 2-answers.

What problems would it cause?

  • Not marking duplicate content as such.
  • Adding irrelevant search results to the 99.9% case of searching for the question.
  • Giving validity to those endless, almost forum-style question threads that we're on a mission to rid the Internet of.
  • Spending time to work on (and risking Google punishing us for) solving a non-issue.
share|improve this answer
    
Very, very insightful! Maybe related: are the "hottest", "newest", "recently active", "unanswered" and "highest voted" results in Google simply caused by Google crawling the site, or do they have some special search merit too? (Whenever my searches are too specific, I often get loooong pages of references to such --outdated-- questions lists in the search results, instead of references to questions. Probably as then my combined search terms are only found in combined, unrelated question summaries that happened to be on the same "hottest '..' questions" list, some day.) –  Arjan Apr 30 '12 at 8:41
    
Interesting, I hardly ever see that these days. I would think that in such a case, Google would usually give you the particular question page, not a list that just contains the title. Do you use Google while being logged in? May they noticed you love SO and desperately try to give you results from there :) Examples would certainly be interesting. –  balpha Apr 30 '12 at 9:50
    
I've been browsing through my history before posting the above, but I guess I mostly end up there at work -- and I'm not there right now. And yes, Google knows I love SE as I use site:stackoverflow.com ;-) Not the best example, but: mvn repo disable ssl site:stackoverflow.com shows a few too. I'll post here, if I find a better one. But so far I did assume that my searches were just bad; mostly cases where I did not really know where to start searching... –  Arjan Apr 30 '12 at 10:36
    
Besides the one problem you wrote this would solve, there's another one which I didn't explicitly mention in my post, since it's not an acute issue yet: we're currently using rel=canonical in a way which Google has explicitly said is improper and not supported, and which can be (and AFAIK has been) used by black hat SEOs to game search results. Yes, there's a (small) risk, as always, that we may lose ranking if we change something; however, there's also a (small) risk that we may lose ranking if we keep doing the same old wrong thing and Google eventually decides to get tougher on it. –  Ilmari Karonen Apr 30 '12 at 10:44
1  
@IlmariKaronen "which Google has explicitly said is improper and not supported" -- No; as I explained, we're not even the use case that Google talks about, and as I explained as well, we're using rel=canonical to point out duplicate content, which is precisely what it's there for. –  balpha Apr 30 '12 at 10:56
    
The online store case seems pretty analogous to me; yes, we have questions and answers instead of categories and products, but to Google they're all just text. Anyway, if you're worried that rel=next/prev won't work properly for our use case, we could a) just ask Google, or b) provide a "view all" page and point the canonical URLs to it, which would solve the problem without any risk of duplicate content or irrelevant search results (and which is what Google recommends as the preferred solution anyway). –  Ilmari Karonen Apr 30 '12 at 11:03
    
Ps. I just looked at the robots.txt link you gave and realized that we have a much bigger SEO problem there. (All the short permalinks we generate for sharing point to disallowed URLs.) But that's for another post, let's not discuss it here. –  Ilmari Karonen Apr 30 '12 at 11:06
    
@IlmariKaronen That last one is a great point, thanks for noting. –  balpha Apr 30 '12 at 14:19
    
@balpha, searching for webkit disable retina sprites just gave me Newest 'retinadisplay' Questions - Stack Overflow as the first hit on Google.nl, and as the second hit on Google.com. Still not an example with many hits like that, and also a bad search query I guess. Still, just in case it helps. –  Arjan May 1 '12 at 10:53
    
@Arjan Hmm yeah, that one actually makes some sense... At least that's filtered by tag; I thought you were talking about the unfiltered active/hot/etc. view –  balpha May 1 '12 at 11:37
    
Yes, you might be right that those "hottest", "newest", ... most often link to tag pages. Those tags might not always be in my search term though; see the long list at site:stackoverflow.com webkit disable retina sprites, which incidentally also shows Newest Questions - Page 37273 (!) on page 3. Unless I report back later, I guess most are tag results indeed, which I understand is okay to you. (Though not useful for me, as often outdated when I click such link.) –  Arjan May 1 '12 at 21:50
    
@Arjan: Oh, wow. That "page 37273" thing does look like a silly thing to index, particularly since it will be stale just as about as soon as it's indexed. Perhaps we should add Disallow: /questions?page= to robots.txt? (I assume we have proper sitemaps etc. so that Google will find our posts without having to keep recrawling all those pages.) Would probably improve crawl performance too... –  Ilmari Karonen May 2 '12 at 12:48
    
@Ilmari, indeed. I reported it to Nick in some comment on a post about that very robots.txt. –  Arjan May 2 '12 at 16:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .