What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

I've been playing with loading the data dumps into MySQL using C#, but even my best attempts still take hours to complete. This question indicates that it can be done in under 10 minutes when pushing the data into MS SQL.

I've commented out the cmd.ExecuteNonQuery() statement which issues the INSERT query and that knocks the time down to just under two minutes. I'm not sure what I'm doing wrong with the database, so I'm wondering if it's just differences between MySQL and MSSQL that are causing the delay, or if it's something I'm doing really, really wrong (I'm assuming the latter is more likely).

I already have a question on StackOverflow regarding the best way to load this data, but it's not exactly getting any attention (hopefully the bounty helps things a bit). What I'm wondering is if there are any statistics on "good" load times using other databases. I'd imagine a large number of folks on meta will have played with this data using a variety of different platforms. Perhaps we could compile some statistics for this?

share|improve this question
4  
Maybe you need more fibre in your diet? –  random Sep 24 '09 at 8:35
    
This is on a local machine right? –  random Sep 24 '09 at 8:53
    
Yeah, everything is on my laptop. Should I have posted this on SO-Proper? I flip-flopped a couple times before posting this here. –  AgentConundrum Sep 24 '09 at 8:59
4  
Sounds like a valid programming question that just happens to use the SO data dump as the basis. –  random Sep 24 '09 at 9:01
    
I threw the code up on my site, after I removed some personal data. Keep in mind this was meant as a quick POC, not production code: appscanadian.ca/LoadMySQL.rar –  AgentConundrum Sep 24 '09 at 9:14
1  
fair enough.. I guess the question already exists there though, as noted in this question. I think I was really just looking for someone to tell me that things really should be running faster than this, and meta made sense since others would have done similar things with the same data. I guess I'll just wait out the other question and hope someone can help me out there. Thanks anyway. –  AgentConundrum Sep 24 '09 at 9:14
    
Just wait around for the rest of the people to wake up who have actually processed the data dump. –  random Sep 24 '09 at 9:16

1 Answer 1

Agent, you need to use a bulk copy strategy and stream the data into the database bypassing checks.

Executing statements via a command is a slow and painful death.

I am currently getting ~12,000,000 records into SQL Server in less than 5 minutes with this:

Do you want to quickly and easily import the SO Data Dump into your SQL Server?

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .