Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Bulk data insertion into MySql

by CountZero (Bishop)
on Sep 02, 2011 at 16:25 UTC ( #923880=note: print w/ replies, xml ) Need Help??


in reply to Bulk data insertion into MySql

But your code does not use DBIx::BulkLoader::MySql at all! You are using straight DBI.

The fastest option for your bulk loading is probably using the LOAD DATA INFILE statement from MySQL.That statement expect a file with the data and does a very fast insert into the database. You can use Perl to transform your data into a format expected by the LOAD DATA INFILE statement.

CountZero

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James


Comment on Re: Bulk data insertion into MySql
Re^2: Bulk data insertion into MySql
by sowraaj (Novice) on Sep 03, 2011 at 12:38 UTC

    Dear friend,

    When i use 'DBIx::BulkLoader::MySql' i got the error that shows @INC...., .pm file not find. I noted that it may due to http://code.activestate.com/ppm/DBIx-BulkLoader-Mysql/. if you don't mine explain the LOAD DATA INFILE concept in a small program(Now i insert 10,00,000rows/3min)I would like to reduce that time to seconds

    Thanks

      I doubt that even the LOAD DATA INFILE statement will insert 10 lakh records in a few seconds.

      As far as DBIx::BulkLoader::Mysql is concerned, did you try installing it through cpan?

      CountZero

      A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://923880]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2014-10-01 11:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (9 votes), past polls