Possible Duplicate:
Etiquette of Screen-scraping Stack Overflow?

Considering that there isn't yet a defined api or even a release date for it, I've resorted to using some cURL requests to scrape content off of the various {M}S[O,U,F] sites.

I'm currently setting a unique USERAGENT

curl_setopt ($ch, CURLOPT_USERAGENT, "[M]S[OUF] - http://codebase.unchartedmess.com");

and not doing any heavy scraping. Mostly it's just the front page looking for updates and tagged questions, but it could end up actively crawling the sites.

Is there a polite way to do this?

  • 1
    Have you taken a look at the data dump?
    – Jon Seigel
    Commented Apr 8, 2010 at 3:34
  • I'm not particularly worried about past data as I am about current data. Right now I'm working on a local project that will scroll current activity from the various trilogy sites all on one page.
    – Josh K
    Commented Apr 8, 2010 at 3:43


Browse other questions tagged .