Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Parallel processing with ForkManager

by DrHyde (Prior)
on Sep 20, 2012 at 10:42 UTC ( #994628=note: print w/ replies, xml ) Need Help??


in reply to Parallel processing with ForkManager

Parallel::ForkManager is certainly a good tool for managing a bunch of processes all under the control of a single "master" process which, in your case, would be the one that reads the 100MB file. However, you need to be careful.

Things to consider include:

  • How many parallel clients can the database handle before it becomes a significant bottleneck?
  • What is the overhead of forking - it's almost certainly too high to naively fork a new process for processing each line in the file.
  • What do you need to do with the data retrieved from the db? While Parallel::ForkManager can return data from each forked process, it fakes this up by going via the disk. Will this turn into an I/O bottleneck?
  • What is the overhead of connecting to the DB, and how can you reduce that?


Comment on Re: Parallel processing with ForkManager

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://994628]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2014-12-22 06:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (111 votes), past polls