Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Perl script speed

by hippo (Curate)
on May 30, 2014 at 08:31 UTC ( #1087955=note: print w/ replies, xml ) Need Help??


in reply to Perl script speed

In any optimisation task the first step is to profile the code. I guess that you have not yet done this (otherwise why would you not have shared the results with us).

Fortunately perl has a plethora of profiling options. The first port of call is the recommended Devel::NYTProf. Alternatively you could try the minimalist Devel::FastProf.

Once you've profiled it you will be in a better position to know where the problem lies.

In the meantime I would suggest breaking out your big nest of if/elses into separate subroutines for ease of both profiling and general maintenance.


Comment on Re: Perl script speed
Re^2: Perl script speed
by rr27 (Initiate) on May 30, 2014 at 09:48 UTC
    As evident, this is first time I am writing perl with SQLite, hence the inefficiency. I have done the profiling and got 1.DBI::st::execute as major time consumer. 2. DBD::SQLite::st::_prepare 3.DBI::_setup_handle. Please suggest ways to reduce the time.
      I would have expected that parsing a 34 MB file would not take very much time (at least not anything near 30 minutes) and that the DB inserts would likely be the big time consumers (but it is of course much better to check it with profiling tools). Well, davido has suggested some possible improvements here.

      Try switching AutoCommit off and then use $dbh->commit() after say every 1000 inserts, adjusting the number to get the best result.
      This node might help Loading bulk data into SQLite

      poj

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1087955]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (6)
As of 2014-12-28 18:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (182 votes), past polls