Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

How to create thread pool of ithreads

by elf_firein (Novice)
on Jan 13, 2009 at 10:56 UTC ( #735923=perlquestion: print w/replies, xml ) Need Help??
elf_firein has asked for the wisdom of the Perl Monks concerning the following question:

I want to create pool of ithreads which will be used on demand.After completion of job it will go back to pool & can be taken back from pool when required.Is to possible to use the available Thread::Pool module for pooling?Or any other way to create thread pool of ithreads? With Regards, Kaustuv

Replies are listed 'Best First'.
Re: How to create thread pool of ithreads
by BrowserUk (Pope) on Jan 13, 2009 at 11:42 UTC

    You have (at least) 3 options:

    1. If you like huge, complicated, overengineered and slow, go with Thread::Pool.

      Ask for help when you run into problems.

    2. Start with this complete and working skeleton and fill-in/replace the lines below ## comments like this ## to suit your application:
      #! perl -slw use strict; use threads; use Thread::Queue; my $n = 100; sub getWorkItems { ## return next workitem ## return $n--; } sub worker { my $tid = threads->tid; my( $Qwork, $Qresults ) = @_; while( my $work = $Qwork->dequeue ) { my $result; ## Process $work to produce $result ## $result = "$tid : result for workitem $work"; $Qresults->enqueue( $result ); } $Qresults->enqueue( undef ); ## Signal this thread is finished } our $THREADS = 10; my $Qwork = new Thread::Queue; my $Qresults = new Thread::Queue; ## Create the pool of workers my @pool = map{ threads->create( \&worker, $Qwork, $Qresults ) } 1 .. $THREADS; ## Get the work items (from somewhere) ## and queue them up for the workers while( my $workItem = getWorkItems() ) { $Qwork->enqueue( $workItem ); } ## Tell the workers there are no more work items $Qwork->enqueue( (undef) x $THREADS ); ## Process the results as they become available ## until all the workers say they are finished. for ( 1 .. $THREADS ) { while( my $result = $Qresults->dequeue ) { ## Do something with the result ## print $result; } } ## Clean up the threads $_->join for @pool;
    3. Assuming this is related to perl ithread issue, post code, cut-down to just demonstrate the problem you are having, and help us to help you. </li

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Hi, My application will continuously fill up the QWork object. Within certain interval it will fill up the QWork object. I want to reuse the threads you have created in pool in every interval it is filling up the QWork object. I am giving you some code snippet: &<code> while (1) { sleep(10); ##execute the stored procedure## ##filling up the QWork object## } <code>& How to do it?expecting your help.
        Actually I want to send ithreads to wait-state when they complete their works.I want take those threads back from wait-state when their are data to get processed by threads.Can thr->signal can be used to put the threads to wait state? Please send me your advices.
        Yep....I am anonymous monk.Sorry for replying late as I was busy with other works. Thanks for your siggestion. It has worked. I need to do one thing.....I need to create a pool of DBI connection. I tried to use Thread::Queue to create create thread-safe Queue for DBI connection. But it is throwing some error like something "Thread is unable to tie" those DBI connection variables. How to create DBI connection pool? I am planning to create an array of DBI connections & planning to use Thread::Semaphore. Will it be good idea? or any other readymade trick? Waiting for your suggestion.
Re: How to create thread pool of ithreads
by Corion (Pope) on Jan 13, 2009 at 11:03 UTC

    Thread::Pool seems to be the module for that. Personally, I do that through Thread::Queue, by having one producer thread and multiple consumer threads from the queue.

Re: How to create thread pool of ithreads
by gone2015 (Deacon) on Jan 13, 2009 at 16:18 UTC

    It may or may not be obvious that using a thread from a "pool" is different from creating a new thread, in particular where not-shared variables are concerned...

    Creating a new thread clones the state of the creator. That includes the state of all not-shared variables. The new thread's arguments can include references to things, even if they are not shared -- in which case the new thread receives references to its own copies of the not-shared things. You can, therefore, pass objects and other complex data structures (including file handles) to a new thread -- essentially on a pass-by-value basis (changes made by the thread do not affect the creator's copies of these things).

    When passing stuff to an existing thread, using Thread::Queue as suggested by others, the simple approach is to enqueue a single scalar, which will be picked up by a worker thread to tell it what to do. That can be:

    1. a simple scalar, into which the dispatcher can pack whatever it wants. To pass file/socket handles the trick is to pass the file number. When unpacking the data the worker thread can rebuild and rebless objects; for file handles it can dup the file number.

    2. a reference to a shared variable (scalar, array or hash), in which case the reference is passed directly -- but it's worth checking whether a given object can be shared ! See "BUGS AND LIMITATIONS" in threads::shared.

    3. until relatively recently the above were the only options. With recent Thread::Queue, however, you can enqueue a reference to a not-shared variable, in which case it will be cloned (recursively) into a shared variable and a reference to the shared version is passed.

      The last change relevant to this facility appears to be v2.08 14-May-2008. Where objects are concerned this is subject to the same "BUGS AND LIMITATIONS".

      This means that you can prepare an argument list and pass it to the worker thread (by enqueuing a reference to that list) in much the same way as creating a new thread and passing arguments to it. Which is something of a step forward, IMHO -- though I have not tested it to destruction.

Re: How to create thread pool of ithreads
by zentara (Archbishop) on Jan 13, 2009 at 14:37 UTC

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://735923]
Approved by Corion
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (5)
As of 2017-02-23 21:06 GMT
Find Nodes?
    Voting Booth?
    Before electricity was invented, what was the Electric Eel called?

    Results (351 votes). Check out past polls.