5

I've been webscraping the chat room with as goal to retrieve some statistics. (have a look at this meta on security.SE (What do you guys want to know about what is said in the DMZ?)

After indexing all the messages, it turns out I'm missing a bunch messages. I'm not sure why I'm missing these messages, so I've got two questions:

  1. Is there an easier way to get the complete dataset rather than scraping the chatroom?
  2. How are these messages counted? Does an edit count as a new message? Do deleted messages still count?

I'm off by about 15 ~ 20 percent.

My source code is here: https://github.com/cloud101/StackExchangeChatScraper

Background

I'm currently loading all the messages in Elasticsearch which I can then query with Apache Spark. The goal is to get some really dumb, but funny statistics which I will display on a seperate site using React.js (this whole thing started with me wanting to learn react.js and thinking of a fun project).

Improve this question
9
  • What is your starting point to scrape?
    – Shadow Wizard
    Commented Sep 29, 2015 at 11:03
  • My starting point is chat.stackexchange.com/transcript/151/2010/12/15
    – Lucas Kauffman
    Commented Sep 29, 2015 at 11:06
  • And how you go forward via the days? If only by "next day" link it's wrong, since some days have several "hour range", e.g. here you have "01:00-13:00" and then "13:00-14:00" and if you miss those, you lose quite a lot of messages indeed.
    – Shadow Wizard
    Commented Sep 29, 2015 at 11:10
  • I actually do have the hour messages as well. I extract them from the pager header and add them to the process queue. In total I have about 495.000 messages indexed, but there should be around 650.000.
    – Lucas Kauffman
    Commented Sep 29, 2015 at 11:19
  • I suspected the same thing though, so I verified it for a random day which had the hours and the message was present in the set.
    – Lucas Kauffman
    Commented Sep 29, 2015 at 11:21
  • The one other thing which might be is that I'm overwriting the message id. But I'm assuming the message id of a chat message within the HTML is "unique".
    – Lucas Kauffman
    Commented Sep 29, 2015 at 11:27
  • Heh, what's with the repeated comment?
    – M.A.R.
    Commented Sep 29, 2015 at 11:28
  • Hmm... Out of ideas then.
    – Shadow Wizard
    Commented Sep 29, 2015 at 11:28
  • Any chance you're hitting the rate limit for http requests? Space out your requests a little.
    – John Dvorak
    Commented Oct 2, 2015 at 9:42

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .