Problems? Is your data what you think it is? | |
PerlMonks |
Re^3: ithreads or fork() what you recommend?by BrowserUk (Patriarch) |
on May 19, 2012 at 14:38 UTC ( [id://971421]=note: print w/replies, xml ) | Need Help?? |
Based on the detailed explanation you have given, I feel that multiprocessing using fork() would be more appropriate then threads. I thought of using threads due to only one reason - it would have avoided significant code change and still I would have benefited by parallel processing of sub tasks. Hm. Nothing in the sparse details you've outlined gives me cause to reach that conclusion; especially if -- as you've suggested -- using fork would require a substantial re-write. Let's say the basic structure of your current serial application is something like:
Then converting that to concurrency using threads could be a simple as:
But I see parallel processing giving around 40-50% (10 Hours) reduction in overall processing time, hence this question. Given the capacity of the hardware you have available, I could well see the above reducing the runtime to less that 5% of the serial version; though the devil is in the details you have not provided. Of course, using Parallel::ForkManager should allow a very similar time reduction, using a very similar minor modification of the existing code. Why you feel that using fork should require a substantial re-write is far from obvious from the scant details you've provided. Ditto, the need for file-based counting and locking. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
In Section
Seekers of Perl Wisdom
|
|