|Do you know where your variables are?|
DBI::SQLite slownessby Endless (Beadle)
|on Sep 20, 2013 at 01:15 UTC||Need Help??|
Endless has asked for the
wisdom of the Perl Monks concerning the following question:
It was suggested to me that I use a database to help prune out duplicate entries (based on 12-digit id numbers) in data I'm processing. I had previously been using a hash, but it starts to choke by 55 million entries. So I fired up DBI::SQLite to give it a try. And I was shocked by what I found.
I simply must have something wrong, because inserting into an sqlite database seems to be occurring at a rate of only 5 per second for me; just writing raw to hashes was going between 11,000 and 20,000 per second. This is unbelievably, unusably slow. What is going on?