Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re: Parallel processing with ForkManager

by DrHyde (Prior)
on Sep 20, 2012 at 10:42 UTC ( #994628=note: print w/replies, xml ) Need Help??

in reply to Parallel processing with ForkManager

Parallel::ForkManager is certainly a good tool for managing a bunch of processes all under the control of a single "master" process which, in your case, would be the one that reads the 100MB file. However, you need to be careful.

Things to consider include:

  • How many parallel clients can the database handle before it becomes a significant bottleneck?
  • What is the overhead of forking - it's almost certainly too high to naively fork a new process for processing each line in the file.
  • What do you need to do with the data retrieved from the db? While Parallel::ForkManager can return data from each forked process, it fakes this up by going via the disk. Will this turn into an I/O bottleneck?
  • What is the overhead of connecting to the DB, and how can you reduce that?
  • Comment on Re: Parallel processing with ForkManager

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://994628]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (5)
As of 2018-03-19 03:08 GMT
Find Nodes?
    Voting Booth?
    When I think of a mole I think of:

    Results (232 votes). Check out past polls.