Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re^2: Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)

by tphyahoo (Vicar)
on Oct 24, 2006 at 18:12 UTC ( #580349=note: print w/ replies, xml ) Need Help??


in reply to Re: Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)
in thread Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)

But it seems that this will only work to run command line programs in parallel with open3.

I want something "functional" -- eg, something that will let me run a series of functions (which in perl I'm reading "code references") in parallel, like goog does with MapReduce.

I'm thinking maybe something more along the lines of Parallel::Simple.

Thanks all the same, because that was a very interesting read.


Comment on Re^2: Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)
Re^3: Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)
by tilly (Archbishop) on Oct 24, 2006 at 21:57 UTC
    Yes, that only works with command line programs.

    But it is pretty easy to convert function calls to command line programs.

    In any case, I wouldn't use Parallel::Simple because I have always valued the ability to control how many children I have at once. (Generally there is a fairly fixed amount of potential parallelism. Using that many kids get gets maximum throughput. If you go higher or lower your overall throughput goes down.) Therefore if you really don't want to convert function calls then I'd use something like Parallel::ForkManager and build my own solution.

      >> Therefore if you really don't want to convert function >> calls then I'd use something like Parallel::ForkManager >> and build my own solution.

      I did this at Using DBM::Deep and Parallel::ForkManager for a generalized parallel hashmap function builder (followup to "reducing pain of parallelization with FP")

      The difficulty was getting my mapped result set back into a hashref that I could return at the end of the function, after all my child processes finished. I wound up "re agglomerating" my hash by storing the results of each function call on the hard drive with DBM::Deep.

      I wound up needing to use a separate DBM::Deep file for each element of the hash, as I was unable to do this with a single DBM::Deep file, although I attempted to take advantage of locking support.

      I wonder if there are other ways, including perhaps your previous suggestion to use ipc open3, perhaps bypassing having to store stuff on the hard drive completely. to be continued...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://580349]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (5)
As of 2014-09-21 20:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (176 votes), past polls