Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

Having the parent (that doesn't build the data nor do any of the tasks) "dole out" tasks to the children (for the one child) adds a lot of complexity vs. just having a pipe for the children to read from. It also can easily reduce concurrency. The significant increase in complexity comes from this one process trying to manage competing tasks that then need to be done asynchronously.

To keep the code simple, you could have the parent just be responsible for spawning children. One child builds the hash and sends out jobs on a pipe that all of the other children read (fixed-length) jobs from. Those children then write their responses that aren't larger than "the system buffer" to a "return" pipe using one syswrite().

This leads to only one tiny bit of "async" fiddling to worry about in only the one child and in a way that can be handled simply in Perl while also only requiring two simple pipes.

But to be able to have children pick up tasks from a single pipe, you have to use fixed-length task data (not larger than the system buffer, 4KB on Linux). To be able to use a single pipe for responses from workers, responses need to be written in chunks (with a fixed-length prefix that specifies the length of the chunk that must total no more than the system buffer size) using a single syswrite(). (And not all worker systems require a response to be sent back to the holder of the huge hash.)

There might be a bit of complexity added by this "just spawns children" parent needing to keep handles open to both ends of both pipes if we expect it to replace both types of children. I'd probably just have to write the code to be sure of the consequences of that. I'd probably simplify that a bit by having the death of the "build the big hash" child cause the parent to re-initialize by execing itself.

There could still be some added complexity just from having one more process that needs to keep handles to one end of each of the two pipes open. But I think that would only impact the "shut down cleanly" code. Again, until I swap all of the little details in (probably by just writing the code), I'm not sure of the full consequences. (Update: Yeah, the complexity comes because, if workers send back responses, you'll need to add a way for the one "build the tasks" child to tell the parent that it is time to shut down or else a way for the parent to tell the one child "all workers are finished".)

(And getting this approach to work reliably on Windows is mostly a whole separate problem because you can't just use fork() and because Windows pipes don't have the same guarantees -- though you can use "message read mode", which should work but is just a separate problem to solve.)

- tye        

In reply to Re^2: Design advice: Classic boss/worker program memory consumption (less async) by tye
in thread Design advice: Classic boss/worker program memory consumption by shadrack

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others having an uproarious good time at the Monastery: (4)
    As of 2018-06-19 21:02 GMT
    Find Nodes?
      Voting Booth?
      Should cpanminus be part of the standard Perl release?

      Results (115 votes). Check out past polls.