Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?

by unlinker (Monk)
on Aug 16, 2009 at 15:14 UTC ( #789022=note: print w/ replies, xml ) Need Help??


in reply to Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?

Thanks both. On some further work, I now have another question (should really have added to the original question): While I call

$pm->finish;
the callback "run_on_finish" seems to be called much later. Is there some way to get Parallel::ForkManager to call run_on_finish very soon after or as soon as finish is called? Thank you once again


Comment on Re: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
Download Code
Re^2: Should I call waitpid when using Parallel::ForkManager to fork in an infinite loop?
by ikegami (Pope) on Aug 16, 2009 at 17:03 UTC

    run_on_finish is called in the parent process immediately after reaping the child (to get its exit code). It surprises me that it would be called "much later" since the parent is constantly checking if a child has ended if you use the formula in the documentation.

    The only reason it wouldn't be responsive is if you have slow code that executes in the parent. This would delay start getting called, which would delay checking if children have ended. Note that start will actually reap multiple children if more than one have ended.

      Allow me to post the complete code to explain what I see happening. The context is a worker application that watches a beanstalk message queue, forks a child to handle the request and immediately goes back to watching the queue. Let me paste the code and then describe what I see:
      #!/usr/bin/perl use strict; use warnings; use Beanstalk::Client; use Parallel::ForkManager; my $clnt = Beanstalk::Client->new({ server => '127.0.0.1:11300', debug => 1, }) || die "Cannot Connect to Queue Manager"; my $pmgr = Parallel::ForkManager->new(20); $pmgr->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; if ($clnt->delete($exit_code)) { print "Deleted Job $exit_code\n"; } else { print "Error Deleting Job $exit_code: " . $clnt->error . " +\n"; } } ); $pmgr->run_on_start( sub { my ($pid, $ident) = @_; if ($clnt->bury($ident)) { print "Buried Job $ident\n"; } else { print "Error burying Job $ident: " . $clnt->error . "\n"; } } ); INFINITE_LOOP: while (1) { sleep 5; my $job = $clnt->reserve(); #blocks until a message is found in qu +eue if (! $job) { print "Error Reserving - Possible Deadline Approaching: " . $c +lnt->error ."\n"; next INFINITE_LOOP; } print "Reserved Job " . $job->id . "\n"; $pmgr->start($job->id) and next; sleep 120; # Job work would go here $pmgr->finish($job->id); } exit;
      When I run this code, here is what I find: (1) When the messages are coming at intervals of say 10 seconds, run_on_finish is never called. (2) After the 10th or 12th message or so, suddenly (no pattern that I can discern) run_on_finish is called repeatedly to reap all 10 (or 12) processes that have ended. (3) Again for about 10/12 messages, no run_on_finish and then suddenly again after 10/12 messages all finished tasks are reaped. Would appreciate if you could help me understand whats going on

        Would appreciate if you could help me understand whats going on

        I've already explained:

        The only reason it wouldn't be responsive is if you have slow code that executes in the parent.

        In addition to intentionally slowing down the parent (sleep 5;), your parent blocks (my $job = $clnt->reserve();). This prevents it from reaping in a timely manner.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://789022]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (8)
As of 2014-12-22 13:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (117 votes), past polls