Re^7: Thread::Pool and Template Toolkit

[BrowserUk]

I've had several attempts at installing Thread::Pool, both using CPAN.pm and manually, and I cannot persuade Thread::Tie to pass it test suite amongst other failures. Maybe that's a win32 thing, but maybe not.

However, I'd have to conclude that Thread::Pool is not ready for prime time. And unless liz pops her head up to say otherwise, as far as I know, she is no longer actively developing her raft of Thead::* modules.

That said, I've never found the need for this level of abstraction with threads. Creating and managing a pool of threads is so easy to do natively--usually 5 to 10 lines of code--that I would probably never use (or create) a tool to do this for me.

Please do not construe this to mean that threads are broken or unusable. That would be like downloading Math::BigInt::Roman, having problems with it, and declaring that Perl can't count.

Since 5.8.3, I've found iThreads to be extremely stable and usable, though they do require a little thought and care to use well (no surprise there). On my platform, I would say they are production ready, but your milage may vary.

[perldragon80]

I agree, it shouldn't be that hard to create my own throttling and pool managing code, but since I am a thread newbie I haven't been able to come up with anything as quick and clean as using the module. I have noticed that without the module I can fire off up to about 10 threads at once without seeing the system run into performance type issues, but anything after that and the actions start taking longer than if there were no threads involved and the thread function (in this case telneting to a router using Net::Telnet::Cisco and getting stats) has a tendency to timeout. Example of what I am doing with Thead::Pool:

*In my case @values is a list of IP addresses usually between 25-30 total values.

my $pool = Thread::Pool->new( {workers => 5, do => \&telnet2Cli} );

my $count = 0;      
foreach $value (@values) {
               $thr[$count] = $pool->job($threadUse, $value, "show ver
+sion");
               $count += 1;                                
}
                          
my $index;
my @versionOutput;
for ($index=0; $index < $count; $index++) {
            $versionOutput[$index] = $pool->result( $thr[$index] );   
+                  
}

[BrowserUk]

Well, the following does the equivalent (of the bit you have shown) without using Thread::Pool.

my $Qin = new Thread::Queue;
my $Qout = new Thread::Queue;

my @pool = map{ 
    threads->new( \&telnet2Cli, 'show version' )->detach 
} 1 .. 5;

my $running : shared = @pool;
$Qin->enqueue( @values );

sleep 1 while $running;

my @versionOuput = $Qout->dequeue;

Of course, that requires that the telnet2Cli sub be written to use the $Qs, but that highlights a problem with Thread::Pool. It isn't a Pool!

With a pool, the members of the pool loop over the worklist processing items until there are no more and then die. That is to say, only 5 (in this case) threads are ever created. These are re-used until the work is finished.

From your snippet (though not confirmed by from the module source, as it is so complex it is hard to follow), I conclude that Thread::Pool, creates a new thread for each work item. The workers => 5 parameter simply restricts the number of concurrent threads to 5 at any given time, but ultimately, one thread will have been created for every value in @values.

Starting ithreads is not cheap, and starting a new thread for every item throws away most of the benefits of using threads.

[perldragon80]

Thank you for your help!
I am definitely getting closer. After playing around with Thread::Queue for a while I have created my telnet2cli function as follows. I think I am missing how it works exactly since the program hangs after 5 worker iterations and doesn't do any more than that. Do I need to return the resultQ from the function? Also, as an aside, being a thread and Thread::Queue newbie, I thought if I detached the threads than I would lose any return values, which I definitely need.

Following from your code I have:


my $Qin = new Thread::Queue;
my $Qout = new Thread::Queue;

my @pool = map{ 
                threads->new( \&telnet2Cli, $Qin, 'show version')->det
+ach 
            } 1 .. 5;

my $running : shared = @pool;
$Qin->enqueue( @values );

sleep 1 while $running;

my @versionOuput = $Qout->dequeue;




sub telnet2Cli {
            require Net::Telnet::Cisco;
             
            my ($ServerQueue, $command) = @_;
            my @output = ();
            my $output = '';
            
            my $ResultQ = new Thread::Queue;
            my $Server = $ServerQueue->dequeue;
            
            my $session = Net::Telnet::Cisco->new(Host => $Server, inp
+ut_log => "input.log.$Server");
            $session->login(Password => 'Password here'); 
            if ($session->enable("Password here")) {
                        $session->cmd("terminal length 0");
                        @output = $session->cmd($cmd);        
                        $output .= "@output";   
                        
                        print "Output about to be enqueued to ResultQ 
+is $output\n";
                        $ResultQ->enqueue($output);
            } else {
                        warn "Cant enable: " . $session->errmsg;
            }
            $session->close;
            
            return $ResultQ;              
}


When I run the program from the command line I do see the expected output from the print statement, but it never continues beyond the 5th connection.