Is there any reasonable guidelines for how much/often I can screen-scrape from SO?
I've been looking around heavily in related topics and have seen Jeff's enthusiastic response to the user information grabber made in Python, a post of someone implementing this program to scrape the user profile of all SO users, but I also seem to remember reading a blog(?) post somewhere by Jeff who seemed quite upset by people who contributed heavy traffic to the site (someone who used a .NET application that made several requests per second, and Yahoo's Web crawler).
I'm interested in creating an application that allows offline/online browsing, with most of the functionality of SO, and some of my own ideas. Of course, with no API available, I'm forced to screen-scrape for the content, and at more regular intervals than the several types of rep-trackers out there since I require up-to-date post/comment information as well.
I've considered scraping all the content with my server and serving it out from there, but that would create the illusion that all the requests are coming from one source and returning to one, which isn't the case - and I wouldn't want to be blacklisted for such a reason.
On the other hand, I can make the requests come from the users themselves, but this would actually cause more traffic, although it would not be able to be traced back to me.
Are there any guidelines to follow, or posts by the SO team for me to read as to how to do this in an acceptable manner, or should this idea be abandoned? I'm unaware of how much traffic I'd actually generate, but I would hate to have my work reduced to nothing by getting blocked if people actually use my application.