Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

Re: Perl script speed

by hippo (Abbot)
on May 30, 2014 at 08:31 UTC ( #1087955=note: print w/replies, xml ) Need Help??

in reply to Perl script speed

In any optimisation task the first step is to profile the code. I guess that you have not yet done this (otherwise why would you not have shared the results with us).

Fortunately perl has a plethora of profiling options. The first port of call is the recommended Devel::NYTProf. Alternatively you could try the minimalist Devel::FastProf.

Once you've profiled it you will be in a better position to know where the problem lies.

In the meantime I would suggest breaking out your big nest of if/elses into separate subroutines for ease of both profiling and general maintenance.

Replies are listed 'Best First'.
Re^2: Perl script speed
by rr27 (Initiate) on May 30, 2014 at 09:48 UTC
    As evident, this is first time I am writing perl with SQLite, hence the inefficiency. I have done the profiling and got 1.DBI::st::execute as major time consumer. 2. DBD::SQLite::st::_prepare 3.DBI::_setup_handle. Please suggest ways to reduce the time.

      Try switching AutoCommit off and then use $dbh->commit() after say every 1000 inserts, adjusting the number to get the best result.
      This node might help Loading bulk data into SQLite

      I would have expected that parsing a 34 MB file would not take very much time (at least not anything near 30 minutes) and that the DB inserts would likely be the big time consumers (but it is of course much better to check it with profiling tools). Well, davido has suggested some possible improvements here.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1087955]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (8)
As of 2018-02-21 19:24 GMT
Find Nodes?
    Voting Booth?
    When it is dark outside I am happiest to see ...

    Results (287 votes). Check out past polls.