Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re^2: DBI::SQLite slowness

by Endless (Beadle)
on Sep 20, 2013 at 12:23 UTC ( #1055013=note: print w/replies, xml ) Need Help??

in reply to Re: DBI::SQLite slowness
in thread DBI::SQLite slowness

Brilliant! With that little fix, my speed is up to 2022 per second; that's almost workable, and I understand what was happening. Now time to start looking through the other suggestions.

Replies are listed 'Best First'.
Re^3: DBI::SQLite slowness
by vincent_veyron (Sexton) on Sep 20, 2013 at 16:20 UTC
    This is what I get on an Atom eee pc (1.6Ghz), after I removed 
    use v5.16.0;
    and changed 
    say "Total time: ", (time - $start); # 180 seconds 
    print "Total time: ", (time - $start); # 180 seconds
    time perl
    Total time: 5
    real	0m5.348s
    user	0m0.360s
    sys	0m0.820s
     : Gestion des contrats, des dossiers contentieux et des sinistres d'assurance
      I've never heard reports that 5.16.0 will significantly slow a program, or that say is so much slower. What's going on here?
Re^3: DBI::SQLite slowness
by Laurent_R (Canon) on Sep 21, 2013 at 09:33 UTC

    Well, 200 millions records at a rate of 2000 per second, that's still 100,000 seconds, or almost 28 hours. That's still pretty long, isn't-it? Having said that, you may be able to live with that, a full day of processing is still manageable for a number of cases. Beware, though, that the rate might slow down as you database grows larger.

    If you are really only looking for filtering out duplicates, the ideas discussed by BrowserUk are probably much better than using a database.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1055013]
[virtualsue]: I missed my 17 year anniversary
[erix]: a virtualanniversary , seems apt enough :P
[marto]: :(
[Discipulus]: we can remedy: ♪♫ Happy monkday to you, Hapymonkday virtualsue..♫..
hippo waves at the other 6
NodeReaper sees 10 souls
shmem sees a creepy one

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (7)
As of 2018-04-24 11:05 GMT
Find Nodes?
    Voting Booth?