I want to make a usable chat search and in order to do this I want to gather the chat messages from SO.
AFAIK chat is not in the data dump nor is it available through the API. So I want to crawl the chat transcripts for messages.
As a test I will start with the transcript of room 11 by starting at the present and working my way back to the first message. Once I have done that (and everything is ok) I will only need to crawl for updates instead of the entire thing.
Currently the transcript contains 1209 days * 5 time frames == 6045 pages. I want to crawl pages with 10 seconds interval (throttling) to get all messages within a single day.
I found this 2 years old meta post about the same thing and I am wondering:
- whether that still applies
- whether you SO people are fine with me crawling the pages with above interval
- whether I can expect problems (i.e. unexpected throttling from your side)
