13

I just updated my Stack Overflow community dump importer:

Changes:

  • It works around a few tiny glitches in the dump (empty user names)
  • It splits up the Tags and creates a PostTags table
  • It loads up all the new columns in the August dump

Due to the increased amount of work (tag splitting) the total time to run the import has gotten a little out of control, it now takes 8 minutes instead of 6 minutes (on my machine tm) to load up the data dump.

If anyone knows of a faster way to get all this info into SQL Server please let me know.

Improve this question
2
  • any chance you could add the compiled app? i don't know the slightest thing about C# but i'd still like to import the data into sql server
    – Kip
    Commented Aug 3, 2009 at 20:00
  • Download is available here: github.com/SamSaffron/So-Slow/downloads
    – waffles
    Commented Aug 4, 2009 at 0:24

4 Answers 4

Reset to default
3

Updated:

https://meta.stackexchange.com/questions/45333/fast-multi-platform-data-dump-import-sql-2000-05-08-sqlite-mysql

This one is pretty fast(er) ;-P

Improve this answer
0
2

It seems like you're at a good pace, it takes me about 12 minutes to import everything with PHP into MySQL, where I import each table in parallel (the longest table takes 12 minutes, others are finished in 1-8 minutes or so depending on length). It also involves applying a regular expression to every line of the XML files. I haven't optimized the script yet, which I plan on doing tonight; but don't really expect too much performance gains. (If anyone uses PHP and can do this faster let me know!)

The only suggestion I have is, have you tried running in parallel? The database should allow multiple connections, and I don't expect each instance of the program to consume 100% of the available processor, but I don't have experience with .NET at all.

Improve this answer
1
  • yeah, that makes sense I should be able to load up the tables in parallel, it may give a little boost, though stuff is pretty much IO bound. When I split up the tags I actually have to use 3 concurrent connections, one for getting tagids another for selecting unsplit tags and another for inserting the data
    – waffles
    Commented Jul 8, 2009 at 22:01
0

If you load the data without any manipulation at all it takes 6 minutes, but if you perform data manipulation it adds 2 minutes? I don't see that as a dramatic amount of additional time spent. Unless you're concerned about the overall time it takes, and not just the addition of the data manipulation. In that case we would need some performance statistics on where your bottleneck is. That will tell you where to focus.

Improve this answer
0

Excel churns endlessly, so 8 minutes is pretty good by comparison! :-D

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .