Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
it is too large to share in mass via this site.

I do have email... but if you can't show me, I cannot help.

From a program architecture standpoint, I am having to do a lot of low level thread and thread pool management in my code. As I progressed from writing the first job step to the 7th, I have developed a module that encapsulates this pattern; however, it isn't work that I am proud of or want to support. I have been through a number of thread pool management modules on CPAN. Most simply don't work, don't fully work, or don't work well with the current versions of perl.

Re: The highlighted portion. This is a myth! (Or simply beginners coding.)

Thread pool management only becomes complex or laborious when people insist on trying to wrap it up in an API. It is the very process of that wrapping that creates complexity. There are so many ways to use thread pools, that writing a wrapper has to deal with so many possibilities that it just creates complexity everywhere. Which is why I don't use such modules.

Written the right way, thread pools don't need to be 'managed', they manage themselves. Here is a complete working example in 30 lines, some of which are just for show. I've typed it so many times now, I can do it from memory, and get it right first time, every time:

#! perl -slw use strict; use Time::HiRes qw[ time ]; use threads; use Thread::Queue; sub worker { my $tid = threads->tid; print "Thread: $tid started"; my $Q = shift; while( my $workitem = $Q->dequeue ) { print "Thread: $tid processing workitem $workitem"; sleep $workitem; } print "Thread: $tid ended"; } our $WORKERS //= 10; our $ITEMS //= 1000; our $MAXWORK //= 2; my $Q = new Thread::Queue; my @workers = map threads->create( \&worker, $Q ), 1 .. $WORKERS; for ( 1 .. $ITEMS ) { sleep 1 while $Q->pending > $WORKERS; $Q->enqueue( rand( $MAXWORK ) ); print "main Q'd workitem $_"; } print "Main: telling threads to die"; $Q->enqueue( (undef) x $WORKERS ); print "Main waiting for threads to die"; $_->join for @workers;

A run:

I think I have come up with an approach

Okay, it looks like you are decided and there's nothing left for me to try and help you with. Good luck.

I'll finish by saying this though. I bet that if you'd give me the specs for your problem so that I could write the thread-pool version, it would be quicker and simpler to write, easier to maintain, and scale better than your event-driven approach.

(That's a bold claim given I have no knowledge of what your code does. But hey, you gotta live a little :)


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?


In reply to Re^3: thread/fork boss/worker framework recommendation by BrowserUk
in thread thread/fork boss/worker framework recommendation by learnedbyerror

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (5)
As of 2024-04-23 01:10 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found