Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: Database input speed question

by Corion (Pope)
on Jul 31, 2003 at 15:24 UTC ( #279611=note: print w/replies, xml ) Need Help??

in reply to Database input speed question

The simple answer is, don't use DBI. Every decent database has a specialized "loader" program, that bypasses SQL completely and pumps in the raw data for each table from a data file. The various loader programs vary in syntax and features, but for Oracle, have a look at sqlloader, Sybase and MSSQL have a similar program.

The problem is now that you can't have computed fields in your database, as the loader will most likely not have the capabilities, but you can always load the database first and then update the fields that need (Perl) computation.

perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The $d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider ($c = $d->accept())->get_request(); $c->send_response( new #in the HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web

Replies are listed 'Best First'.
Re: Re: Database input speed question
by CountZero (Bishop) on Jul 31, 2003 at 19:29 UTC

    The loader may not have the intelligence to do computed fields, but Perl does. As most likely you will have to write a Perl script anyhow to reformat your data into a format acceptable to the bulkloader, you can take that opportunity to fill the computed fields.


    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://279611]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2018-11-14 09:22 GMT
Find Nodes?
    Voting Booth?
    My code is most likely broken because:

    Results (165 votes). Check out past polls.