Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Re: Question about Parallel::ForkManager

by Anonymous Monk
on Oct 02, 2011 at 12:31 UTC ( #929140=note: print w/ replies, xml ) Need Help??

in reply to Question about Parallel::ForkManager

If you spawn one process for each URL, you're bound to cause massive thrashing and this is where your disk-usage is coming from. Instead, launch a small number of child processes who consume URLs from a thread-safe queue. The number of children should have no relation to the size of the workload that they must cooperatively accomplish. Rather, it should be tied to how many parallel processes you have determined the system can actually handle with maximum sustained throughput. (Do not be surprised if the best answer is "1.")

Comment on Re: Question about Parallel::ForkManager

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://929140]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2015-10-09 16:49 GMT
Find Nodes?
    Voting Booth?

    Does Humor Belong in Programming?

    Results (247 votes), past polls