Your skill will accomplish
what the force of many cannot
using parallel processing to concatenate a string, where order of concatenation doesn't matterby tphyahoo (Vicar)
|on Oct 18, 2006 at 10:48 UTC||Need Help??|
tphyahoo has asked for the
wisdom of the Perl Monks concerning the following question:
I am experimenting with using threading, or forking, or whatever you want to call it, to speed things up that are slow, but where order of execution doesn't matter.
Here we have an artificial example, where I want to concatenate some letters, but the order of concatenation doesn't matter.
I thought I could use Parallel::ForkManager to achieve my aim like in the following, but as the tests demonstrate, this doesn't work.
The sub concatenate_serial does what I want, but is slow. It takes 9 seconds total, because I slowed things artifically with sleep(3). The other method, concatenate_parallel, finishes in 3 seconds, but it doesn't concatenate the letters.
I think I am probably making an error in thinking, or in misunderstanding what threads, or forks, or whatever you call it, is supposed to work.
I see that's the problem. What I don't have is a way to share these variables across the memory process. Well, I suppose one way would be to write the concat string to a permanent store on the hard drive, like a file or a db. But I'm wondering if there's a more elegant way, something that works "just with ram".
ANOTHER UPDATE: Okay, so I guess threads works "just with ram". Could someone throw some code my way that I could plug in to convert my serial code into parallel code (well, to be real pedantic, parallel code that can't be relied on execute sequentially)?
Please help me understand this!
I withdraw to my little cell now for further study and meditation.....
UPDATE: I changed the title to "using parallel processing to...." where originally it was "using Parallel::ForkManager to...". Seems like that module doesn't do what I want, but hopefully something with threads does.
UPDATE 2: Now I'm wondering if I could do this with MapReduce, pipin' fresh on cpan... Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?