Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Re^8: Why do my threads sometimes die silenty?

by alain_desilets (Beadle)
on Sep 22, 2011 at 11:33 UTC ( #927328=note: print w/replies, xml ) Need Help??

in reply to Re^7: Why do my threads sometimes die silenty?
in thread Why do my threads sometimes die silenty?

Well, what I need to do is a bit more complicated than that ;-).

Basically, I have a very complex object of class WeBiText. The object carries out a complex search algorithm, part of which, called the L1 Search, does a number of searches on Yahoo, that produces hits to be processed by the rest of the search algorithm. Because the Yahoo searches take a long time to execute, I want the L1 search to be done in a separate thread, and in such a way that it communicates the hits it finds back to the main search algorithm as soon as it finds them. That way, the main search doesn't have to wait for the complete set of L1 hits before it starts doing its work.

There are issues that complicates things.
  1. WeBiText is not thread safe, because it needs some modules which are not thread-safe.
  2. A consequence of this is that all threads in the thread pool must be created before any instance of WeBiText is created.
  3. Another consequence is that WeBiText instances can't be shared between threads and must instead be serialized and deserialized across threads.
  4. Also, I run hundreds of automated tests on WeBiText, using PerlUnit
A consequence of points 2 and 4, is that threads must be "reusable", otherwise I would have to create hundreds of threads at the start, before I run the tests which create WeBiText instances. So the pool must allow a client to "reserve" a thread and later on release it so it can be reused. This is one functionality that Thread::Pool provides.

Also, a consequence of point 3 is that the L1 search cannot directly notify the WeBiText instance running in the main thread when it finds a new L1 hit, because it doesn't receive a shared instance of that WeBiText instances. The best that can be done is for the L1 thread to receive a serialization of the master thread's WeBiText instance, and use that serialization to recreate the instance in the L1 search thread. But any change to the state of that recreated instance won't affect the state of the original instance running in the main thread.

My solution to this is for the L1 search to put its hits on an "outgoing" queue that is shared between the two threads. The problem is that both threads must know about this queue. But since WeBiText instances must be created after the threads are created, it means that any queue that it creates cannot be shared with those threads. So, it's the pools responsibility to provide a results queue that is associated with a particular thread. Those hits queues must be reserved and released just like the threads.

Note that this behaviour is not provided by Thread::Pool, and I need to implement it myself anyway. But since Thread::Pool already provides reusable threads, I thought I'd start from it. Maybe that was a mistake.

As you can see, this is not something that can be implemented in a couple of lines ;-).
  • Comment on Re^8: Why do my threads sometimes die silenty?

Replies are listed 'Best First'.
Re^9: Why do my threads sometimes die silenty?
by BrowserUk (Pope) on Sep 22, 2011 at 11:56 UTC
    There are issues that complicates things. ...

    Oh. So you need bidirectional communications with your threads.

    That complicates things -- but not so much:

    #! perl -slw use strict; use threads; use Thread::Queue; sub worker { my( $Qwork, $Qback ) = @_; my $tid = threads->tid; print "Thread $tid started"; while( defined( my $work = $Qwork->dequeue ) ) { print "Thread $tid processing workitem $work"; $Qback->enqueue( $work * $tid ); sleep 1; } $Qback->enqueue( undef ); print "Thread $tid done"; } our $WORKERS //= 10; our $ITEMS //= 30; my $Qwork = new Thread::Queue; my $Qback = new Thread::Queue; my @pool = map threads->new( \&worker, $Qwork, $Qback ), 1 .. $WORKERS +; $Qwork->enqueue( 1 .. $ITEMS ); $Qwork->enqueue( (undef) x $WORKERS ); for( 1 .. $WORKERS ) { while( defined( my $res = $Qback->dequeue ) ) { print "Main retrieved result: $res"; } } print "Main done"; $_->join for @pool

    A run:

    But I sense that you will not be satisfied with simple, so good luck :)

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://927328]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2018-06-18 10:37 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (109 votes). Check out past polls.