Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: DBI::SQLite slowness

by Jenda (Abbot)
on Sep 20, 2013 at 09:17 UTC ( #1054984=note: print w/ replies, xml ) Need Help??


in reply to DBI::SQLite slowness

When a hash based solution starts to choke because of the amount of data, the easiest solution is to use an on-disk hash. See BerkeleyDB or DB_File. The only change necessary will be to load the module and tie the hash. The rest of the script will stay the same.

Jenda
Enoch was right!
Enjoy the last years of Rome.


Comment on Re: DBI::SQLite slowness
Re^2: DBI::SQLite slowness
by Laurent_R (Parson) on Sep 21, 2013 at 09:24 UTC

    I tried that a couple of time, the performance on inserting new records just became really bad. It may be OK when the DB file already exists and is more or less static, but apparently not when you are constantly adding new records. It may also be that I did not do it the right way, but I tried various things, I don't see what I might have done wrong.

      It will always be quite a bit slower than when the hash fits in memory, but still much quicker than if the hash grows so big that the computer has to swap memory pages onto disk.

      Jenda
      Enoch was right!
      Enjoy the last years of Rome.

        Agreed.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1054984]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (7)
As of 2014-10-01 11:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (10 votes), past polls