Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^3: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?

by unlinker (Monk)
on Aug 16, 2009 at 18:03 UTC ( #789038=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
in thread Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?

Allow me to post the complete code to explain what I see happening. The context is a worker application that watches a beanstalk message queue, forks a child to handle the request and immediately goes back to watching the queue. Let me paste the code and then describe what I see:

#!/usr/bin/perl use strict; use warnings; use Beanstalk::Client; use Parallel::ForkManager; my $clnt = Beanstalk::Client->new({ server => '127.0.0.1:11300', debug => 1, }) || die "Cannot Connect to Queue Manager"; my $pmgr = Parallel::ForkManager->new(20); $pmgr->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; if ($clnt->delete($exit_code)) { print "Deleted Job $exit_code\n"; } else { print "Error Deleting Job $exit_code: " . $clnt->error . " +\n"; } } ); $pmgr->run_on_start( sub { my ($pid, $ident) = @_; if ($clnt->bury($ident)) { print "Buried Job $ident\n"; } else { print "Error burying Job $ident: " . $clnt->error . "\n"; } } ); INFINITE_LOOP: while (1) { sleep 5; my $job = $clnt->reserve(); #blocks until a message is found in qu +eue if (! $job) { print "Error Reserving - Possible Deadline Approaching: " . $c +lnt->error ."\n"; next INFINITE_LOOP; } print "Reserved Job " . $job->id . "\n"; $pmgr->start($job->id) and next; sleep 120; # Job work would go here $pmgr->finish($job->id); } exit;
When I run this code, here is what I find: (1) When the messages are coming at intervals of say 10 seconds, run_on_finish is never called. (2) After the 10th or 12th message or so, suddenly (no pattern that I can discern) run_on_finish is called repeatedly to reap all 10 (or 12) processes that have ended. (3) Again for about 10/12 messages, no run_on_finish and then suddenly again after 10/12 messages all finished tasks are reaped. Would appreciate if you could help me understand whats going on


Comment on Re^3: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
Download Code
Replies are listed 'Best First'.
Re^4: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
by ikegami (Pope) on Aug 16, 2009 at 21:43 UTC

    Would appreciate if you could help me understand whats going on

    I've already explained:

    The only reason it wouldn't be responsive is if you have slow code that executes in the parent.

    In addition to intentionally slowing down the parent (sleep 5;), your parent blocks (my $job = $clnt->reserve();). This prevents it from reaping in a timely manner.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://789038]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (16)
As of 2015-07-31 15:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (279 votes), past polls