Re^8: Why do my threads sometimes die silenty?

[alain_desilets]

Well, what I need to do is a bit more complicated than that ;-).

Basically, I have a very complex object of class WeBiText. The object carries out a complex search algorithm, part of which, called the L1 Search, does a number of searches on Yahoo, that produces hits to be processed by the rest of the search algorithm. Because the Yahoo searches take a long time to execute, I want the L1 search to be done in a separate thread, and in such a way that it communicates the hits it finds back to the main search algorithm as soon as it finds them. That way, the main search doesn't have to wait for the complete set of L1 hits before it starts doing its work.

There are issues that complicates things.

1. WeBiText is not thread safe, because it needs some modules which are not thread-safe.
2. A consequence of this is that all threads in the thread pool must be created before any instance of WeBiText is created.
3. Another consequence is that WeBiText instances can't be shared between threads and must instead be serialized and deserialized across threads.
4. Also, I run hundreds of automated tests on WeBiText, using PerlUnit

A consequence of points 2 and 4, is that threads must be "reusable", otherwise I would have to create hundreds of threads at the start, before I run the tests which create WeBiText instances. So the pool must allow a client to "reserve" a thread and later on release it so it can be reused. This is one functionality that Thread::Pool provides.

Also, a consequence of point 3 is that the L1 search cannot directly notify the WeBiText instance running in the main thread when it finds a new L1 hit, because it doesn't receive a shared instance of that WeBiText instances. The best that can be done is for the L1 thread to receive a serialization of the master thread's WeBiText instance, and use that serialization to recreate the instance in the L1 search thread. But any change to the state of that recreated instance won't affect the state of the original instance running in the main thread.

My solution to this is for the L1 search to put its hits on an "outgoing" queue that is shared between the two threads. The problem is that both threads must know about this queue. But since WeBiText instances must be created after the threads are created, it means that any queue that it creates cannot be shared with those threads. So, it's the pools responsibility to provide a results queue that is associated with a particular thread. Those hits queues must be reserved and released just like the threads.

Note that this behaviour is not provided by Thread::Pool, and I need to implement it myself anyway. But since Thread::Pool already provides reusable threads, I thought I'd start from it. Maybe that was a mistake.

As you can see, this is not something that can be implemented in a couple of lines ;-).

[BrowserUk]

There are issues that complicates things. ...

Oh. So you need bidirectional communications with your threads.

That complicates things -- but not so much:

#! perl -slw
use strict;
use threads;
use Thread::Queue;

sub worker {
    my( $Qwork, $Qback ) = @_;
    my $tid = threads->tid;
    print "Thread $tid started";
    while( defined( my $work = $Qwork->dequeue ) ) {
        print "Thread $tid processing workitem $work";
        $Qback->enqueue( $work * $tid );
        sleep 1;
    }
    $Qback->enqueue( undef );
    print "Thread $tid done";
}

our $WORKERS //= 10;
our $ITEMS   //= 30;

my $Qwork = new Thread::Queue;
my $Qback = new Thread::Queue;

my @pool = map threads->new( \&worker, $Qwork, $Qback ), 1 .. $WORKERS
+;

$Qwork->enqueue( 1 .. $ITEMS );
$Qwork->enqueue( (undef) x $WORKERS );

for( 1 .. $WORKERS ) {
    while( defined( my $res = $Qback->dequeue ) ) {
        print "Main retrieved result: $res";
    }
}
print "Main done";

$_->join for @pool

A run:

<Reveal this spoiler or all in this thread>

But I sense that you will not be satisfied with simple, so good luck :)