2

I am working on some analysis on Stack Exchange data, and I want to download the Posttags table in CSV or XML format. But the problem is, this table is not available in the archive, and it has a large number of records and query compose (check first link) only retrieves 50K records so I am not able to use the "Download CSV" option. Is there a way I can download the whole table data through SQL or setting a no-limit option?

1 Answer 1

4

You can run multiple queries to get the data and then join the CSVs together. For example, grab data for the PostIds between 1 and 10,000 first (max of 5 tags per post, so that's guaranteed to be under the 50K limit), then 10,001 to 20,000, etc. There's currently 98,900 posts on Mathematica available through the data explorer, so this shouldn't take too long.

If you're interested in which tags are on which post, you can get that info from Posts.xml in the data dump. What are you actually trying to do with the data, though? There may be a better way.

5
  • I am dumping this data into MySQL for my academic project. To design prototype, I am using Mathematica data but in future I want to use StackOverflow data which is huge as compared to Mathematica. So, in that case, this manual operation will not help me as it will not be a feasible option. Nov 8, 2015 at 5:03
  • 1
    The Posttags table can be recreated from the posts.xml. The tags field holds the tags, enclosed in < and >. So you can parse that, lookup the id for each tag (as you get those in tags.xml) and insert the found tagid and postid into the posttags table.
    – rene
    Nov 8, 2015 at 9:11
  • @SiMemon Check out rene's suggestion in the comments here. That'd be my recommendation for a more scalable solution here.
    – Adam Lear StaffMod
    Nov 8, 2015 at 20:09
  • @rene Thanks a lot. I used python to get the data and create the new table as you suggested. Worked like a charm !! Nov 9, 2015 at 6:26
  • @AdamLear Thanks for your valuable suggestions!! Nov 9, 2015 at 6:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .