Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Fork Results in thousands of processes

by sundialsvc4 (Abbot)
on Nov 29, 2011 at 15:48 UTC ( #940646=note: print w/ replies, xml ) Need Help??


in reply to Fork Results in thousands of processes

Modules such as Parallel::ForkManager might be useful.

CPAN modules like these are “simple sugar,” to be sure, in that they do not do anything that you could not do for yourself, but they do make life very easy ... they let you focus more on what you want to do, and less on exactly how to do it.   The example in the documentation is virtually identical to what you are trying to do here.

You might also wish to search http://search.cpan.org for modules like Thread::Queue.   (There are, at the moment, 33 such modules that are found by that search.)   Because it may well be that you want to start a limited number of processes (say, ten ...) and then to have each of those processes consume an arbitrary number of work-requests that you have provided for them in a queue.   In other words, by launching (say...) ten workers, you ensure that ten requests at a time will be handled simultaneously; and, by stuffing (say...) 1,000 names or commands into a queue, you provide the total list of commands that will be carried out, cooperatively, by the workers in that thread-pool.   Stuff the queue full of filenames, followed by enough “please die now” instructions for all of the workers to eventually receive one.   Then, simply wait for them to die off on-command.

The nice thing about a design like this, is that it has a very convenient throttle.   You can set the number of worker threads up or down, and this has nothing to do with the amount of work that is to be accomplished.   Even if the backlog grows very large indeed, the completion rate will remain steady and predictable and can be effectively “tuned” to suit the hardware.


Comment on Re: Fork Results in thousands of processes
Re^2: Fork Results in thousands of processes
by BrowserUk (Pope) on Nov 29, 2011 at 17:06 UTC

    How are you going to use Thread::Queue which uses shared process memory with Parallel::ForkManager which creates multiple processes?

    Why do you keep blurting out these pointless, useless, whole wrong replies to questions on subjects you obviously have no practical knowledge whatsoever?

    Do you just hate newbies? Or just delight on sending them off down blind alleys for your own amusement?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      The two comments are obviously not intended to be so closely related, mine unintended enemy, as anyone who is not “mine unintended enemy” would very plainly understand.   (Do you downvote my posts both as yourself and using some sock-puppet account?   Just curious.)

      There are two very-valid points contained in the aforementioned reply:

      1. That you don’t need to futz around with too-much manually written code ... and the bugs and annoyances that go with it, e.g. in the OP’s original solution ... because CPAN is well-equipped with solutions that other folks have already written.
      2. That it is very useful, in any multi-thread or multi-process solution, to use “work to do” queues, with a limited number of workers who consume entries from the queue.   The number of workers is independent from the number of requests that need to be serviced.   An existing CPAN package is employed to implement the queue and all of its perhaps operating-system-specific ugliness.

      What, then, is “the point?”   Easy.   At first, you approach this language with the notion that “I have to write this.”   Then, you discover that, thanks to CPAN, you don’t “have to write this.”   You do not have to stumble-around with code that is literally a take-off of Unix fork() when there is, perhaps unbeknownst to you, a package like Parallel::ForkManager at your beck and call.   That’s huge.   But not obvious.

      Quite frankly, good sir, your ongoing determination to conduct a vendetta against me clouds your own vision.   I think that even “newbies” are likely to be professional computer programmers who are very able indeed to know when a particular package, cited only as a “for instance,” is or is not appropriate to their particular project.   Yeah, I’ll just betcha that they do know that “threads and processes play by different rules in most operating systems,” without either you or I having any moral obligation to educate them concerning this point.   So perhaps we should all simply trust the newbies on that score, as peculiar as that thought may initially seem.   I will wager that neither you nor I are, in fact, “the smartest kids in this school.”   And, yeah, I am perfectly aware that you have earned many times more experience-points than have I, so I am not presuming that this discussion is “among equals.”

        (Do you downvote my posts both as yourself and using some sock-puppet account?

        No. I don't use sock puppets. I am quite happy to have my actions accounted for.

        Which means at least one other person recognises your crap for what it is and feels strongly enough to take action.

        For the rest. Nothing but noise. You really think an experienced programmer fork bombs themselves and has to ask why it happened.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Re^2: Fork Results in thousands of processes
by Arunbear (Parson) on Nov 30, 2011 at 15:10 UTC
    Did you mean Proc::Queue (which limits the number of forks you can do) ?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://940646]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2014-09-23 04:44 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (210 votes), past polls