Beefy Boxes and Bandwidth Generously Provided by pair Networks DiBona
more useful options
 
PerlMonks  

Re^3: DBI::SQLite slowness

by Laurent_R (Priest)
on Sep 21, 2013 at 09:33 UTC ( #1055120=note: print w/ replies, xml ) Need Help??


in reply to Re^2: DBI::SQLite slowness
in thread DBI::SQLite slowness

Well, 200 millions records at a rate of 2000 per second, that's still 100,000 seconds, or almost 28 hours. That's still pretty long, isn't-it? Having said that, you may be able to live with that, a full day of processing is still manageable for a number of cases. Beware, though, that the rate might slow down as you database grows larger.

If you are really only looking for filtering out duplicates, the ideas discussed by BrowserUk are probably much better than using a database.


Comment on Re^3: DBI::SQLite slowness

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1055120]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (9)
As of 2014-04-16 11:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (424 votes), past polls