I recall a mention by Jeff and Joel, perhaps on the podcast, of forwarding requests from web-crawlers to a dedicated environment.

The idea was to serve the same content as the real site, but to have more control over the format for SEO optimization as well as to keep the crawlers from overwhelming the "live" site.

I'd like to know if there is any additional descriptions of this approach and whether it was just a passing thought or an idea that was implemented.

  • To receive the bounty I'd like to know not only whether/whether-not SO does this, but some basic pros/cons and considerations of the approach – STW May 4 '11 at 22:05
  • 3
    You would be better served for the general question (pros/cons and considerations) by going to webmasters.stackexchange.com – Pollyanna May 4 '11 at 22:54

Search engines are routed through a separate back-end on our HAProxy server - but they point to the same servers and see the same content that anyone else would see when browsing (well except most of them don't implement javascript - but that's not something we do). We do this simply to allow us to have more fine grained control of the resources that web crawlers utilize and to be able to tweak it without effecting real people's experience.

| improve this answer | |
  • 1
    What do you use to detect the spider-ness of a spider? User agent sniffing? IP block ownership? Magic unicorn dust? – Charles May 4 '11 at 23:19
  • We just base it off user agent, not perfect but reliably catches the big guys. – Zypher May 4 '11 at 23:27
  • @Zypher So search engines, like have a standard for their bots or somethin'? – Yatharth Agarwal Sep 11 '12 at 14:51
  • @YatharthROCK no no standard, but the big guys A) have good UA strings, and B) those UA strings are well known. – Zypher Sep 11 '12 at 16:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .