Re^7: Sqlite: Threads and inserts into a different database for each one. (1 thread fast, >1 slow: BY YOUR DESIGN!)

[BrowserUk]

Given that $uniq_values is a hash reference, could you tell me what you think this construct does?

\%{$uniq_values}

And why you think it is necessary?

[Anonymous Monk]

Hum..


now that you put my eyes on it and make me think about it, it seem i was thinking to call my subroutine with a reference but i call it with a reference of an already referenced hash ?

And i have to admit than i don't know the (cpu/memory cost) cost of this behavior ? neither the real consequences of this "wrong design"...

And it seems, after re-reading of some topics, than i try to process some data inside the worker's subroutine without being shared, or push them with enqueue, and the result is a deep copy off the hash(es), array(s) for each threads and maybe the time took to create the thread is here?

I didn't already tried to recreate the multi threads process by splitting my data inside a shell script, and multithread it inside this script to observe if it's really a limitation of multiple sqlite connection to different databases inside a same script or only an incomprehension of the conseption of the threads processes inside perl.

I don't have many experiences with perl (even if i've already worked with it) but i refuse to give up without real facts and good understanding of my errors, and really think than the slowness i talk about on this topic, is not related by the fact than I multithread my insert but than my code to create the pool is the "dark side" of my code shown by the small debugs i've made, or like your first observations considering your "skills" :) (without offense anybody)

Best regards,

[BrowserUk]

now that you put my eyes on it and make me think about it, it seem i was thinking to call my subroutine with a reference but i call it with a reference of an already referenced hash ?

The construct \%{ $hashref } is exactly the same as just $hashref :

%hash = 'a'..'j';;
$hashref = \%hash;;

pp $hashref;;
{ a => "b", c => "d", e => "f", g => "h", i => "j" }

$copy = \%{ $hashref };;
pp $copy;;
{ a => "b", c => "d", e => "f", g => "h", i => "j" }

print for $hashref, $copy;;
HASH(0x394aa70)
HASH(0x394aa70)

My reason for questioning it is simply that when someone uses a very obscure noop quite so frequently in their code, it raises alarm bells. Why are they doing this? What were they hoping to achieve by it?

You are using a large number of other, similar NOOP constructs; eg. \@{ $arrayref }, and whole bunch of scalar references (to scalars holding small strings) that don't appear to serve any other purpose than to obfuscate your code.

None of this has any particular performance bearing on the code, but it has made reading and understanding what your code is doing, a lot harder than it ought to be.

The code has also changed somewhat from the code you originally posted -- eg. in the original, your thread procedures were posting great lumps of data to a results queue:

          $Qresults->enqueue( \%{$result} );

But then nothing is ever read from that queue. That however has disappeared from the code you posted later.

But the big thing I'm having trouble understanding is why you are querying a huge amount of data from one DB, only to create temporary tables in another DB, and then perform a bunch of selects on it.

Why not just create those temporary tables in the original DB and do the selections there?

You are also building up huge volumes of shared data -- much of it duplicated -- and then doing nothing with it.

In general, I'm having a real hard time trying to understand the overall goals of program. It seems to serve no useful purpose other than to use threads and shared data in the worst possible way.

Could you write a textual description of what the goal of this code is?