Re: Multi-threads newbie questions

[BrowserUk]

The solution I'm afraid is "Don't use Thread::Pool or Thread::Pool::Simple". They're broken.

Anything you enqueue (using the ->add() method) gets stringified using Storable, and so by the time your threads get something, it is a frozen/thawed copy of the original. Nothing they do to it will ever be reflected back to the original. They are just horribly, horribly broken.

If you would care to describe your real application, I'd have a go a suggesting an approach to solving it.

[daverave]

Thank you BrowserUk. Perhaps I should use 'Thread::Queue`' then?

In any case, I will describe my application in short, as you requested:

I'm processing many genomes. Each genome is stored in a hash, which includes some basic data about the genome (organism, size etc.) and also many file location (genome sequence etc.). Each genome hash is what I previously referred to as an 'internal hash'. All those hashes are stored together in one big hash.

The 'helper' sub, which we can now call 'process_genome', takes care of a single genome. It does some stuff, including calling external scripts which e.g. convert file formats, and add key-val pairs to the genome hash, e.g. new file locations.

I would like to process all genomes. Since I have 8 cores on my server, I would like to use multi-threading. I would like to give as input a hash of (genome) hashes, and get back a similar structure, but updated.

That's all, I think.

[BrowserUk]

BTW. Always, the easiest, safest way to approach threading complex applications, is to write a single-threaded version that operates upon the data in a serial fashion.

Once you have that working, if the data is truly independent, parallelising it is usually quite simple.

For completeness, here is another example based on yours that uses a pool of threads. It is hardly more complicated than the first version:

#! perl -slw
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Data::Dump qw[ pp ];

sub helper {
    my $Q = shift;
    while( my $ref = $Q->dequeue ) {;
        lock $ref;
        $ref->{NEW_KEY} = 1;
    }
}

sub my_sub {
    my( $ref, $n ) = @_;
    my $Q = new Thread::Queue;
    my @threads = map async( \&helper, $Q ), 1 .. $n;
    $Q->enqueue( values %{ $ref } );
    $Q->enqueue( (undef) x $n );
    $_->join for @threads;
}

my $hoh = {
    A => shared_clone( { NAME => 'aa' } ),
    B => shared_clone( { NAME => 'bb' } ),
};

pp $hoh;
my_sub( $hoh, 2 );
pp $hoh;

The output is identical to the earlier version.

[BrowserUk]

Okay. Here's a very simple example (that works :), based on yours above:

#! perl -slw
use strict;
use threads;
use threads::shared;
use Data::Dump qw[ pp ];

sub helper {
    my $ref = shift;
## Not needed if no more that one thread will access each subhash
##    lock $ref;         
    $ref->{NEW_KEY} = 1;
}

sub my_sub {
    my $ref = shift;
    my @threads = map async( \&helper, $_ ), values %{ $ref };
    $_->join for @threads;
}

my $hoh = {
    A => shared_clone( { NAME => 'aa' } ),
    B => shared_clone( { NAME => 'bb' } ),
};

pp $hoh;
my_sub( $hoh );
pp $hoh;

Output:

<Reveal this spoiler or all in this thread>

[daverave]

Thank you BrowserUk.

Now please allow me some follow-ups.

First, regarding the creation of $hoh. I tried replacing your

my $hoh = {
    A => shared_clone( { NAME => 'aa' } ),
    B => shared_clone( { NAME => 'bb' } ),
};

with

my $hoh = {
    A => { NAME => 'aa' },
    B => { NAME => 'bb' },
};
$hoh = shared_clone($hoh);

Why doesn't this work? doesn't 'shared_clone' does deep sharing?

This is not just a hypothetical question. I get my $hoh unshared (I might even retrieve it), that is, I do not create it at the same time I call this piece of code. So how can I take an unshared $hoh and use it here?

Second, if I understand correctly, you start a thread for each element of hoh. There are usually a few hundreds elements there, so I guess it's not as good idea. That's why I originally wanted yo use a thread pool, until you advised me otherwise.

Thanks again.

[Jenda]

"it is a frozen/thawed copy of the original." ... and therefore the thread is free to use it any way it likes without having to synchronize access and without danger of stepping on another thread's toes.

You've sent the thread some work to do, once its done it will send you the results.

[BrowserUk]

and therefore the thread is free to use it any way it likes without having to synchronize access and without danger of stepping on another thread's toes.

Did you look at the OPs code?

His requirements mean that he has that ability--safe access without synchronisation. And without the need to duplicate through freeze/thaw (twice). That's part of the benefit of using threads.

You've sent the thread some work to do,

No need. It is a thread, It already has access. All you need to do it telll it what to access.

Using threads like processes, is like buying a moped and pedalling it everywhere.