Beefy Boxes and Bandwidth Generously Provided by pair Networks RobOMonk
No such thing as a small change
 
PerlMonks  

Re: unix loops in perl

by i5513 (Monk)
on Oct 26, 2011 at 14:18 UTC ( #933905=note: print w/ replies, xml ) Need Help??


in reply to unix loops in perl

Could be interesting to use pdsh with exec module:

pdsh -w [1-24] -Rexec perl /home/perl/chr.match.pl chr%h.nonCG.out
It will do for you in parallel, so less time is needed (sure in cpan there are modules to make it in perl :) )
Regards,


Comment on Re: unix loops in perl
Download Code
Re^2: unix loops in perl
by moritz (Cardinal) on Oct 26, 2011 at 15:00 UTC
    It will do for you in parallel, so less time is needed

    Without further proof or explanation, I'd assume this statement to be generally false.

      Ok, I should say 'probabbly' less time is needed

      If your proccess doesn't take many resources (cpu/network/memory), then probably it is true, isn't it?

      So you have 30 process running at same time in an computer which is perfectly capable to do its work, the it will be faster than running 30 process one by one in a bucle.

      If you have an algorithm wich consume all (or near to) cpu / memory / network, then it will not be fastest but slower

      Do you agree now ?

      Regards,

        If each process takes up very little resources, then the overhead of creating them will cause it all to go slower.

        If the bottleneck is network or memory, then extra processes won't help.

        If the bottleneck is CPU, then the processes will let you use more cores, so that will help to a point. You probably don't have 30 cores, so a smaller number of them doing more work each would be better.

        If you've got a healthy mix of resources, then a second process could help, allowing one to crunch numbers while the other waits for the disk and then vice versa.

        Like so many things in life, one is good, a few more are convenient, but too much will kill you.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://933905]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (5)
As of 2014-04-21 10:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (493 votes), past polls