2

I recall a mention by Jeff and Joel, perhaps on the podcast, of forwarding requests from web-crawlers to a dedicated environment.

The idea was to serve the same content as the real site, but to have more control over the format for SEO optimization as well as to keep the crawlers from overwhelming the "live" site.

I'd like to know if there is any additional descriptions of this approach and whether it was just a passing thought or an idea that was implemented.

Improve this question
2
  • To receive the bounty I'd like to know not only whether/whether-not SO does this, but some basic pros/cons and considerations of the approach
    – STW
    Commented May 4, 2011 at 22:05
  • 3
    You would be better served for the general question (pros/cons and considerations) by going to webmasters.stackexchange.com
    – Pollyanna
    Commented May 4, 2011 at 22:54

1 Answer 1

Reset to default
6
+50

Search engines are routed through a separate back-end on our HAProxy server - but they point to the same servers and see the same content that anyone else would see when browsing (well except most of them don't implement javascript - but that's not something we do). We do this simply to allow us to have more fine grained control of the resources that web crawlers utilize and to be able to tweak it without effecting real people's experience.

Improve this answer
4
  • 1
    What do you use to detect the spider-ness of a spider? User agent sniffing? IP block ownership? Magic unicorn dust?
    – Charles
    Commented May 4, 2011 at 23:19
  • We just base it off user agent, not perfect but reliably catches the big guys.
    – Zypher
    Commented May 4, 2011 at 23:27
  • @Zypher So search engines, like have a standard for their bots or somethin'?
    – Yatharth Agarwal
    Commented Sep 11, 2012 at 14:51
  • @YatharthROCK no no standard, but the big guys A) have good UA strings, and B) those UA strings are well known.
    – Zypher
    Commented Sep 11, 2012 at 16:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .