Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: Parallel processing with ForkManager

by DrHyde (Prior)
on Sep 20, 2012 at 10:42 UTC ( #994628=note: print w/ replies, xml ) Need Help??

in reply to Parallel processing with ForkManager

Parallel::ForkManager is certainly a good tool for managing a bunch of processes all under the control of a single "master" process which, in your case, would be the one that reads the 100MB file. However, you need to be careful.

Things to consider include:

  • How many parallel clients can the database handle before it becomes a significant bottleneck?
  • What is the overhead of forking - it's almost certainly too high to naively fork a new process for processing each line in the file.
  • What do you need to do with the data retrieved from the db? While Parallel::ForkManager can return data from each forked process, it fakes this up by going via the disk. Will this turn into an I/O bottleneck?
  • What is the overhead of connecting to the DB, and how can you reduce that?

Comment on Re: Parallel processing with ForkManager

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://994628]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (11)
As of 2015-04-01 21:48 GMT
Find Nodes?
    Voting Booth?

    Who makes your decisions?

    Results (50 votes), past polls