Can I crawl stackoverflow articles, questions, answers, user id, tags and so on.
For the purpose of research in data mining, I just need some of corpus in specific period.
2 Answers
Sure! Just make sure you follow the robots.txt file and don't crawl too quickly.
Other options include:
- the data dump, which contains all public data about every Stack Exchange site, including Stack Overflow, updated every two months
- the Stack Exchange Data Explorer, which is updated monthly
- the Stack Overflow API, which has live data (see StackApps for more details)
answered Mar 23, 2011 at 5:42
-
3Actually, given we have the dump and API, why on earth would we want anyone crawl the site at all? Mar 23, 2011 at 9:09
-
@Ivo Well, given that the robots.txt doesn't specifically exclude all crawlers except a select few, and that there is certain information available on the site not on those resources (specifically, related questions), an argument could be made that crawling isn't discouraged. The other places may or may not be better, depending on what various people might want to use the information for. Mar 24, 2011 at 23:32
-
1what do you mean by "don't crawl too quickly"? 1 page per 5 seconds? 1/sec? How fast can one crawl stackoverflow without being impolite (or banned)?– SHoundOct 10, 2011 at 5:06
You can use the API, see here.
