Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

How to tell a child what to do?

by Eyck (Priest)
on Sep 21, 2005 at 09:09 UTC ( #493696=perlquestion: print w/ replies, xml ) Need Help??
Eyck has asked for the wisdom of the Perl Monks concerning the following question:

I need a way to communicate with children (possibly non-blocking and reliable way). 'Spose I do something like this:
  1. I fork of a bunch of children, to await my command
  2. I figure out a tasks that need to be done
  3. I tell each child what needs to be done
  4. When some child dies, I check what was his task, and tell some other available child to do it.

The hard part seems to be communicating with children, with one child, I would probably use STDIN/STDOUT, servers like apache have symmetrical children, so I'm kind of lost as to what is the best way to do something like that.

UPDATE Re^4: How to tell a child what to do? contains lite & correct solution.

UPDATE OK, it seems like merlyn has written a column about the subject, with a line-by-line commentary, which is good. (but overall, I still quite like Anonymous Monk's solution) Re: How to tell a child what to do?

Comment on How to tell a child what to do?
Re: How to tell a child what to do?
by Corion (Pope) on Sep 21, 2005 at 09:12 UTC

    I would use a database as the communication mechanism. That way, you get easy and convenient control over the jobs, when they were started and when they were finished (or hanging). For the job description, you can either come up with a keyword mechanism or consider to put the actual code to be run into the database - this makes for convenient extensibility.

      I'm sorry, but it seems like you're playing with databases and everything looks like a nail.

      Proposing additional and significant external dependencies for a nice self-contained program is not very smart.

        I'm sorry that I misunderstood your problem. I thought that your statement:

        The hard part seems to be communicating with children, with one child, I would probably use STDIN/STDOUT, servers like apache have symmetrical children, so I'm kind of lost as to what is the best way to do something like that.

        implied that your children are not symmetrical and maybe even distributed across machines, and that children die-ing from unknown reasons might be common.

        Maybe you should discuss the scope of your program in the original post to some larger extent - I found DBD::SQLite a very convenient, lightweight and self-contained database, for example.

        You know it might actauly be a nail in some cases. It is odd that you ask a question and then are soo rude when you get an answer. Databases can be the right tool and they might not be. That depends entirely on your situation. If you wanted to have 100 children spread over 100 computers then a database would be a perfect solution. You said you wanted "possibly non-blocking and reliable way". Database is *one* of the answers that fits the question. For instances I work on windows and linux boxes and never feel comforatble with forking or "traditional" IPC. However I am always working with databases and most scripts I write use databases for something (not out of habit, i work with large sets of scheduling and accounting data in a database). In my case lots of times a DB is the easiest, fastest and most comfortable route for my problems. As far as we knew you where already connected to a DB and this would be a good solution.

        "Proposing additional and significant external dependencies for a nice self-contained program is not very smart." -- Considering we don't know the current dependencies of your program, we could hardly deduce that a database wouldn't fit your needs.

        Ask a question, get lots of answers, don't critise an answer or mock the person giving it just because it doesn't fit your need. We can't read your mind or know your particular needs or desires unless you tell us first. Sorry for the rant but its realy hurts to see someone try to help and provide a fair answer and be treated the way you tried corion.


        ___________
        Eric Hodges
Re: How to tell a child what to do?
by Zaxo (Archbishop) on Sep 21, 2005 at 09:33 UTC

    On (4.), what is killing your children before their task is done?

    There are lots of ways to talk to child processes. If there is a small fixed set of messages, signaling with kill is effective, with %SIG copy-on-write. If you need more extensive communication, pipes are my favorite, but sockets are also good. Each child will need to parse what you send it. See perlipc for details.

    Another way is to only launch a child when it and the parent will know what it's supposed to do. That's best on systems, like Linux, where fork is cheap.

    After Compline,
    Zaxo

      A: You (as a designer) don't know.

      Anyhoo, some tasks may take forever, thus you need to keep it running indefinitely.

      I don't know how launching a child only when task is ready is going to help, you still need to tell your newborn child what kind of tasks you prepared for them ( ie, pass some argument, I would go for $ARGV[], this seems simplest ).

Re: How to tell a child what to do?
by GrandFather (Cardinal) on Sep 21, 2005 at 09:35 UTC
    • Ask nicely in a quiet voice
    • Suggest firmly in a modest voice
    • Demand loudly
    • Give up and do it yourself

    Perl is Huffman encoded by design.
Re: How to tell a child what to do?
by Anonymous Monk on Sep 21, 2005 at 09:43 UTC
    The standard way of doing this is using pipe opens. man perlipc has some examples - the example uses communication between the parent and a single child, but filehandles can easily be saved in a datastructure.

    Note that if you use pipe-open, open returns the PID of the child. To determine a child terminating, set up a signal handler to catch SIGCLD. This signal doesn't tell you the pid, but if you do a wait or a waitpid in the handler, you do. Given the PID of the terminated child, you know which task was assigned to it (assuming you stored this information when dealing out the tasks).

    Study man perlipc, it's a real treasure trove. It should have enough examples to get you on your way.

      You are surprisingly right,

      I somehow always regarded

      $pid = open(KID_TO_WRITE, "|-");
      trick as a way to talk with only one child, but if we rewrite it like this:
      $fh = new FileHandle; my $sleep_count = 0; do { $pid = open($fh, "|-"); unless (defined $pid) { warn "cannot fork: $!"; die "bailing out" if $sleep_count++ > 6; sleep 10; } } until defined $pid; my (@some_data)=qw[a be ce]; if ($pid) { print $fh @some_data; close($fh) || warn "kid exited $?"; } else { while (<STDIN>) { print "GOT: $_\n"; # child's STDIN is parent's KID } exit; # don't forget this };
      then we get to hold whole array of different $fd's, each for talking with one child. This would be clean and reliable, exactly what I wanted.

      Thanx.

        on second thought..this doesn't work that well with more then one fd open...it hangs on close($fd). Maybe there's an error in my code:
        print "I'm $$\n;"; use FileHandle; $fh = new FileHandle; $fh1 = new FileHandle; my $sleep_count = 0; do { $pid = open($fh, "|-"); unless (defined $pid) { warn "cannot fork: $!"; die "bailing out" if $sleep_count++ > 6; sleep 10; } } until defined $pid; if ($pid) { print "I'm master($$) $pid...\n"; } else { print "Waiting for commands($$)..\n"; while (<STDIN>) { print "GOT: $_\n"; # child's STDIN is parent's KID } exit; # don't forget this }; do { $pid1 = open($fh1, "|-"); unless (defined $pid1) { warn "cannot fork: $!"; die "bailing out" if $sleep_count++ > 6; sleep 10; } } until defined $pid1; if ($pid1) { print "I'm master($$) $pid1...\n"; } else { print "Waiting for commands($$)..\n"; while (<STDIN>) { print "GOT: $_\n"; # child's STDIN is parent's KID } exit; # don't forget this }; my (@some_data)=qw[a be ce]; print "($$)PID: $pid\n"; print $fh @some_data; print "($$) printing done, let's close:\n"; close($fh) || warn "kid exited $?"; print "($$)PID: $pid1\n"; print $fh1 @some_data; print "($$) printing done, let's close:\n"; close($fh1) || warn "kid exited $?"; print "END FH TEST\n\n\n";
Re: How to tell a child what to do?
by Random_Walk (Parson) on Sep 21, 2005 at 10:56 UTC

    I like using threads and thread queues You can give each child thread its own instruction queue to assign it tasks or have children picking tasks up off a common jobs to be done queue (depends how specialist your children are). The parent can be signalled results back through a common queue for all children. Children can also post heartbeat messages to this queue so you know if one is hanging/dead.

    POE may be worth a look. I had a similar job to do a couple of years ago and POE looked ideal but management refused installation of any (non-core)modules, yawn, so I never got to really use it. Management actualy refused any open source but had tons coded in Perl, go figure.

    Cheers,
    R.

    Pereant, qui ante nos nostra dixerunt!
      actually, threads won't cut it for my problem(to little separation, and reliability), and same goes for POE (because some tasks are blocking).
Re: How to tell a child what to do?
by merlyn (Sage) on Sep 21, 2005 at 14:35 UTC
      It's pretty rare to search my site and not find an article that contains a particular topic

      Oh, Merlyn, you're such a spoil-sport. It is us other Monks which have to say that we're not surprised you have written a column about it!

      ;-)

      CountZero

      "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: How to tell a child what to do?
by QM (Vicar) on Sep 21, 2005 at 17:14 UTC
    I have a script I did a while ago that may help, see Re: Concurrent Processes.

    I'd be interested to know if this is useful for you, or if there are any issues with it. (No one has commented on it in the nearly 2 years it's been posted, so either 1) no one has seen it, 2) it's useful, or 3) it doesn't help, and no one has complained.)

    -QM
    --
    Quantum Mechanics: The dreams stuff is made of

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://493696]
Approved by reasonablekeith
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (16)
As of 2014-09-02 15:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (25 votes), past polls