3

I know we can download the entire Stack Overflow database from the Web Archive

Is there any way to query the Stack Overflow database archive? The entire database is around 30 GB, and I don't want all of it, only the Q&A with a specific tag (or even a combination of two or more tags). For example, say I only want Q&A related to Android programming and I don't want any other categories. Is there a way to download only certain part of database?

3
  • Can you clarify? There are files per each site in the network. If you mean - you just want to get posts in specific tags of a specific site, then no, that's not possible.
    – Oded
    Jul 27, 2016 at 17:40
  • What I want is Q&A related to specific tags. Ex. I want to download all questions and answers having the tag ios from stackOverflow.com Jul 28, 2016 at 5:52
  • You will need to download the complete Posts file, parse out the Tags column for ios and retrieve those posts. Or use the data explorer, as rene answered.
    – Oded
    Jul 28, 2016 at 8:18

1 Answer 1

6

No, you can't do that.

You have to download all the XML before you can do any filtering. If the size of the dump is an issue you might consider The Stack Exchange Data Explorer as it is updated weekly and has a bit more tables. There are some tricks to download lots of data at once for the Data Explorer.

An other option if you don't want to download the whole dump is using the StackAPI but it is limited to 10,000 api calls per day so for huge tags it might take a while before you have all posts in a tag. Start here for the questions, implement paging and fetch the answers-by-id for the questions you found.

Last but not least you could setup some ordinary web-scraping. Watch out for the throttling though.

2
  • There is a app called "Dash" for mac which does this. If its not possible then how are they able to achieve this ? Jul 28, 2016 at 5:49
  • @JafferSheriff I don't know, you have to ask them. But as they are a company I assume they have the resources to download everything, rework it in their format and make selections. An other possibility is that they leverage the StackAPI but they are the only ones who can answer that.
    – rene
    Jul 28, 2016 at 8:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .