What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 133 Stack Exchange communities.

I just updated my Stack Overflow community dump importer:

Changes:

  • It works around a few tiny glitches in the dump (empty user names)
  • It splits up the Tags and creates a PostTags table
  • It loads up all the new columns in the August dump

Due to the increased amount of work (tag splitting) the total time to run the import has gotten a little out of control, it now takes 8 minutes instead of 6 minutes (on my machine tm) to load up the data dump.

If anyone knows of a faster way to get all this info into SQL Server please let me know.

share|improve this question
    
any chance you could add the compiled app? i don't know the slightest thing about C# but i'd still like to import the data into sql server –  Kip Aug 3 '09 at 20:00
    
Download is available here: github.com/SamSaffron/So-Slow/downloads –  waffles Aug 4 '09 at 0:24

4 Answers 4

It seems like you're at a good pace, it takes me about 12 minutes to import everything with PHP into MySQL, where I import each table in parallel (the longest table takes 12 minutes, others are finished in 1-8 minutes or so depending on length). It also involves applying a regular expression to every line of the XML files. I haven't optimized the script yet, which I plan on doing tonight; but don't really expect too much performance gains. (If anyone uses PHP and can do this faster let me know!)

The only suggestion I have is, have you tried running in parallel? The database should allow multiple connections, and I don't expect each instance of the program to consume 100% of the available processor, but I don't have experience with .NET at all.

share|improve this answer
    
yeah, that makes sense I should be able to load up the tables in parallel, it may give a little boost, though stuff is pretty much IO bound. When I split up the tags I actually have to use 3 concurrent connections, one for getting tagids another for selecting unsplit tags and another for inserting the data –  waffles Jul 8 '09 at 22:01

If you load the data without any manipulation at all it takes 6 minutes, but if you perform data manipulation it adds 2 minutes? I don't see that as a dramatic amount of additional time spent. Unless you're concerned about the overall time it takes, and not just the addition of the data manipulation. In that case we would need some performance statistics on where your bottleneck is. That will tell you where to focus.

share|improve this answer

Excel churns endlessly, so 8 minutes is pretty good by comparison! :-D

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .