Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Question about Parallel::ForkManager

by Anonymous Monk
on Oct 02, 2011 at 12:31 UTC ( #929140=note: print w/ replies, xml ) Need Help??


in reply to Question about Parallel::ForkManager

If you spawn one process for each URL, you're bound to cause massive thrashing and this is where your disk-usage is coming from. Instead, launch a small number of child processes who consume URLs from a thread-safe queue. The number of children should have no relation to the size of the workload that they must cooperatively accomplish. Rather, it should be tied to how many parallel processes you have determined the system can actually handle with maximum sustained throughput. (Do not be surprised if the best answer is "1.")


Comment on Re: Question about Parallel::ForkManager

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://929140]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (10)
As of 2014-08-22 11:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (156 votes), past polls