What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 127 Stack Exchange communities.

I got the latest data dump and wrote a PHP script to parse the XML, but the actual parsing is taking forever (the uncompressed Posts.xml is 10GB). In the interim, does anyone know of a MySQL-ready .sql file that can be imported directly into the database?

I googled but the only thing I found was an inactive torrent of a much older version of the data dump, when the data dump was only 250MB. Does anyone know of a new version?

P.S. If anyone can point me to an existing .sql, that would be great. If not, I'll probably upload one when parsing is done, if there's interest. But I'm only parsing the Posts table, as this is all I need.

share|improve this question
To those flagging as off-topic: this question refers to the SE data dump. A perfectly reasonable thing to ask about here on MSO. –  Anna Lear Jun 21 '12 at 3:50
Part of the reason that XML was chosen is because it is a universal format. Your particular request only works with MySQL; SE would have to make a specific file for each database system. –  Robert Harvey Jun 21 '12 at 3:57
Are you using the correct type of parser here? You should be using an event-based parser, not some of the other parser that try to load the entire document into memory. –  Zoredache Jun 21 '12 at 8:31
@Robert Harvey I agree, XML is the best here. I'm not expecting SE to prepare a database-specific file for each database under the sun. That's a lot of work. I was thinking of a community effort to arrange this, as needed, especially for the Posts table. It would be sweet to search questions on the go when no internet connection is available. When I was downloading from clearbits, it said 44 downloaded this file. It's not a lot of people, but maybe half of them use MySQL. Would be nice if we could all save some time. –  Z V Jun 21 '12 at 13:07
@Zoredache Yep, I'm using XMLReader which is the stream-based reader in PHP. I initially tried the other XML readers, but quickly dropped them when they kept crashing the script. I think the reason the script takes a long time is that it's also inserting into the database. –  Z V Jun 21 '12 at 13:07
Disable indexe, queue up a thousand or so posts and do mass inserts. –  Zoredache Jun 21 '12 at 19:52
In case you are unaware of it, there is the Stack Exchange Data Explorer that you can run queries on. Not quite as flexible as having a local copy, but it's pretty cool. –  jadarnel27 Jul 26 '12 at 20:30
All I can say is... don't ever try using Python to parse the Stack Overflow XML files. (Unless you have, say, 24 GB of RAM.) –  uɐɯsO uɐɥʇɐN Dec 23 '12 at 23:54
Heh, I have extra server resources. I could possibly host and generate monthly a ready-to-download database in 1 or 2 formats depending on how well the data compresses –  Earlz Jan 23 '13 at 19:24
Does anyone have a version of the data dump in SQL? –  user1029296 Jun 3 '13 at 15:44
add comment

2 Answers

Here's what I did. Create a script that reads the xml and outputs to a CSV file, and then use LOAD DATA INFILE to load the data into MySQL. Doing inserts one at time will take ages.

Also, I wish the data dump was in CSV to begin with. It's far more universal a format than XML especially when importing into databases is a concern. But I think we've been over that already here.

share|improve this answer
It's far more universal a format than CSV - don't you mean 'than XML'? –  ire_and_curses Jan 23 '13 at 18:24
@ire_and_curses fixed. Thanks. –  Kibbee Jan 23 '13 at 19:15
But CSV is old and tired and XML is all enterprisy and stuff! –  mikeTheLiar Jan 23 '13 at 19:24
add comment

I have used this java code to good effect


.. it runs pretty fast once set up (and you need to add the jdbc driver)

share|improve this answer
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .