I want to make a usable chat search and in order to do this I want to gather the chat messages from SO.

AFAIK chat is not in the data dump nor is it available through the API. So I want to crawl the chat transcripts for messages.

As a test I will start with the transcript of room 11 by starting at the present and working my way back to the first message. Once I have done that (and everything is ok) I will only need to crawl for updates instead of the entire thing.

Currently the transcript contains 1209 days * 5 time frames == 6045 pages. I want to crawl pages with 10 seconds interval (throttling) to get all messages within a single day.

I found this 2 years old meta post about the same thing and I am wondering:

  • whether that still applies
  • whether you SO people are fine with me crawling the pages with above interval
  • whether I can expect problems (i.e. unexpected throttling from your side)


You must log in to answer this question.

Browse other questions tagged .