0

Possible Duplicate:
Etiquette of Screen-scraping Stack Overflow?

Considering that there isn't yet a defined api or even a release date for it, I've resorted to using some cURL requests to scrape content off of the various {M}S[O,U,F] sites.

I'm currently setting a unique USERAGENT

curl_setopt ($ch, CURLOPT_USERAGENT, "[M]S[OUF] - http://codebase.unchartedmess.com");

and not doing any heavy scraping. Mostly it's just the front page looking for updates and tagged questions, but it could end up actively crawling the sites.

Is there a polite way to do this?

marked as duplicate by Bill the Lizard Apr 8 '10 at 3:41

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    Have you taken a look at the data dump? – Jon Seigel Apr 8 '10 at 3:34
  • I'm not particularly worried about past data as I am about current data. Right now I'm working on a local project that will scroll current activity from the various trilogy sites all on one page. – Josh K Apr 8 '10 at 3:43

Browse other questions tagged .