Can I crawl stackoverflow articles, questions, answers, user id, tags and so on.

For the purpose of research in data mining, I just need some of corpus in specific period.

2 Answers 2


Sure! Just make sure you follow the robots.txt file and don't crawl too quickly.

Other options include:

  • 3
    Actually, given we have the dump and API, why on earth would we want anyone crawl the site at all?
    – Ivo Flipse
    Mar 23, 2011 at 9:09
  • @Ivo Well, given that the robots.txt doesn't specifically exclude all crawlers except a select few, and that there is certain information available on the site not on those resources (specifically, related questions), an argument could be made that crawling isn't discouraged. The other places may or may not be better, depending on what various people might want to use the information for.
    – waiwai933
    Mar 24, 2011 at 23:32
  • 1
    what do you mean by "don't crawl too quickly"? 1 page per 5 seconds? 1/sec? How fast can one crawl stackoverflow without being impolite (or banned)?
    – SHound
    Oct 10, 2011 at 5:06

You can use the API, see here.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .