Beefy Boxes and Bandwidth Generously Provided by pair Networks Joe
Just another Perl shrine
 
PerlMonks  

Re^3: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?

by unlinker (Monk)
on Aug 16, 2009 at 18:03 UTC ( #789038=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
in thread Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?

Allow me to post the complete code to explain what I see happening. The context is a worker application that watches a beanstalk message queue, forks a child to handle the request and immediately goes back to watching the queue. Let me paste the code and then describe what I see:

#!/usr/bin/perl use strict; use warnings; use Beanstalk::Client; use Parallel::ForkManager; my $clnt = Beanstalk::Client->new({ server => '127.0.0.1:11300', debug => 1, }) || die "Cannot Connect to Queue Manager"; my $pmgr = Parallel::ForkManager->new(20); $pmgr->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; if ($clnt->delete($exit_code)) { print "Deleted Job $exit_code\n"; } else { print "Error Deleting Job $exit_code: " . $clnt->error . " +\n"; } } ); $pmgr->run_on_start( sub { my ($pid, $ident) = @_; if ($clnt->bury($ident)) { print "Buried Job $ident\n"; } else { print "Error burying Job $ident: " . $clnt->error . "\n"; } } ); INFINITE_LOOP: while (1) { sleep 5; my $job = $clnt->reserve(); #blocks until a message is found in qu +eue if (! $job) { print "Error Reserving - Possible Deadline Approaching: " . $c +lnt->error ."\n"; next INFINITE_LOOP; } print "Reserved Job " . $job->id . "\n"; $pmgr->start($job->id) and next; sleep 120; # Job work would go here $pmgr->finish($job->id); } exit;
When I run this code, here is what I find: (1) When the messages are coming at intervals of say 10 seconds, run_on_finish is never called. (2) After the 10th or 12th message or so, suddenly (no pattern that I can discern) run_on_finish is called repeatedly to reap all 10 (or 12) processes that have ended. (3) Again for about 10/12 messages, no run_on_finish and then suddenly again after 10/12 messages all finished tasks are reaped. Would appreciate if you could help me understand whats going on


Comment on Re^3: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
Download Code
Re^4: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
by ikegami (Pope) on Aug 16, 2009 at 21:43 UTC

    Would appreciate if you could help me understand whats going on

    I've already explained:

    The only reason it wouldn't be responsive is if you have slow code that executes in the parent.

    In addition to intentionally slowing down the parent (sleep 5;), your parent blocks (my $job = $clnt->reserve();). This prevents it from reaping in a timely manner.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://789038]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2014-04-20 17:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (485 votes), past polls