What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 129 Stack Exchange communities.

Can I crawl stackoverflow articles, questions, answers, user id, tags and so on.

For the purpose of research in data mining, I just need some of corpus in specific period.

share|improve this question

migrated from stackoverflow.com Mar 23 '11 at 5:37

This question came from our site for professional and enthusiast programmers.

2 Answers 2

Sure! Just make sure you follow the robots.txt file and don't crawl too quickly.

Other options include:

share|improve this answer
2  
Actually, given we have the dump and API, why on earth would we want anyone crawl the site at all? –  Ivo Flipse Mar 23 '11 at 9:09
    
@Ivo Well, given that the robots.txt doesn't specifically exclude all crawlers except a select few, and that there is certain information available on the site not on those resources (specifically, related questions), an argument could be made that crawling isn't discouraged. The other places may or may not be better, depending on what various people might want to use the information for. –  waiwai933 Mar 24 '11 at 23:32
    
what do you mean by "don't crawl too quickly"? 1 page per 5 seconds? 1/sec? How fast can one crawl stackoverflow without being impolite (or banned)? –  SHound Oct 10 '11 at 5:06

You can use the API, see here.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .