Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re: Sqlite: Threads and inserts into a different database for each one. (1 thread fast, >1 slow)

by sundialsvc4 (Monsignor)
on Apr 08, 2014 at 02:33 UTC ( #1081456=note: print w/ replies, xml ) Need Help??


in reply to Sqlite: Threads and inserts into a different database for each one. (1 thread fast, >1 slow)

In my humble opinion, your problem is still that the various threads will be competing with one another.   Very likely, you do not need threads at all ... very likely, you do not need any of this complexity ... and this just might be what these results are trying to tell you!

I suggest that you would be far better served, either by sticking with the one-process approach which (still) seems to be giving you fairly decent performance, or by modifying the (non-threaded) program so that it scans the input file for the records that belong in a particular database (identified, say, by a command-line parameter ...), and then, if you wish, running multiple instances of this program concurrently say, by using the "&" feature of the Unix/Linux shell.

In the most friendly way possible, I suggest that you’ve built-up all this concurrency-stuff on the expectation that it would save you time, when most-clearly it is not doing so.   And, if an approach is not panning-out, the time comes to just let it go.   Personally, in this case, I think that time has come.


Comment on Re: Sqlite: Threads and inserts into a different database for each one. (1 thread fast, >1 slow)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1081456]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (8)
As of 2014-07-30 10:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (230 votes), past polls