Re^3: thread/fork boss/worker framework recommendation

[BrowserUk]

it is too large to share in mass via this site.

I do have email... but if you can't show me, I cannot help.

From a program architecture standpoint, I am having to do a lot of low level thread and thread pool management in my code. As I progressed from writing the first job step to the 7th, I have developed a module that encapsulates this pattern; however, it isn't work that I am proud of or want to support. I have been through a number of thread pool management modules on CPAN. Most simply don't work, don't fully work, or don't work well with the current versions of perl.

Re: The highlighted portion. This is a myth! (Or simply beginners coding.)

Thread pool management only becomes complex or laborious when people insist on trying to wrap it up in an API. It is the very process of that wrapping that creates complexity. There are so many ways to use thread pools, that writing a wrapper has to deal with so many possibilities that it just creates complexity everywhere. Which is why I don't use such modules.

Written the right way, thread pools don't need to be 'managed', they manage themselves. Here is a complete working example in 30 lines, some of which are just for show. I've typed it so many times now, I can do it from memory, and get it right first time, every time:

#! perl -slw
use strict;
use Time::HiRes qw[ time ];
use threads;
use Thread::Queue;

sub worker {
    my $tid = threads->tid;
    print "Thread: $tid started";
    my $Q = shift;
    while( my $workitem = $Q->dequeue ) {
        print "Thread: $tid processing workitem $workitem";
        sleep $workitem;
    }
    print "Thread: $tid ended";
}

our $WORKERS //= 10;
our $ITEMS   //= 1000;
our $MAXWORK //= 2;

my $Q = new Thread::Queue;
my @workers = map threads->create( \&worker, $Q ), 1 .. $WORKERS;

for ( 1 .. $ITEMS ) {
    sleep 1 while $Q->pending > $WORKERS;
    $Q->enqueue( rand( $MAXWORK ) );
    print "main Q'd workitem $_";
}
print "Main: telling threads to die";
$Q->enqueue( (undef) x $WORKERS );

print "Main waiting for threads to die";
$_->join for @workers;

A run:

<Reveal this spoiler or all in this thread>

I think I have come up with an approach

Okay, it looks like you are decided and there's nothing left for me to try and help you with. Good luck.

I'll finish by saying this though. I bet that if you'd give me the specs for your problem so that I could write the thread-pool version, it would be quicker and simpler to write, easier to maintain, and scale better than your event-driven approach.

(That's a bold claim given I have no knowledge of what your code does. But hey, you gotta live a little :)