Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Re: Introduction to Parallel::ForkManager

by biosysadmin (Deacon)
on Sep 15, 2003 at 05:38 UTC ( #291490=note: print w/ replies, xml ) Need Help??


in reply to Re: Introduction to Parallel::ForkManager
in thread Introduction to Parallel::ForkManager

You are correct, I have used the module just as described in the referenced node. I'm planning on writing up a synopsis of what I had to do, how I did it, and how it all works and stick it under craft.

However, since you asked here's a short answer. I did use Parallel::ForkManager to solve my problem, I followed the following suggestions from the thread:

  • I uncompressed each file to a temporary location (not /tmp) and then deleted it after the reformatting was through.
  • I made sure that the temporary location was on a different physical drive from the permanent location for the formatted files.
  • I rewrote my program to divide the work into 4 distinct sets, and then forked a process for each set.
  • I switched to the GNU version of gzip because of this reply (the GNU version is supposedly faster).
  • I staggered the initiation of the threads by sleeping for 60 seconds to try and avoid competition for disk I/O.
There's a bit more about the optimization that I've done, but look for it under Craft in the next few weeks.


Comment on Re: Re: Introduction to Parallel::ForkManager

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://291490]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2015-07-04 02:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (57 votes), past polls