Possible Duplicate:
Etiquette of Screen-scraping Stack Overflow?
Considering that there isn't yet a defined api or even a release date for it, I've resorted to using some cURL requests to scrape content off of the various {M}S[O,U,F] sites.
I'm currently setting a unique USERAGENT
curl_setopt ($ch, CURLOPT_USERAGENT, "[M]S[OUF] - http://codebase.unchartedmess.com");
and not doing any heavy scraping. Mostly it's just the front page looking for updates and tagged questions, but it could end up actively crawling the sites.
Is there a polite way to do this?
