Beefy Boxes and Bandwidth Generously Provided by pair Networks Joe
laziness, impatience, and hubris
 
PerlMonks  

exec vs. system

by Microcebus (Beadle)
on Jul 07, 2011 at 12:17 UTC ( #913169=perlquestion: print w/ replies, xml ) Need Help??
Microcebus has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have a perlscript called swap.pl an want to start it several times with another perl script to use my 4 CPU cores. Each time, the swap.pl script processes a different part of a given dataset. My code looks like:

foreach(1..$number_of_cpus) { open(TEMP_CTRL,">temp_ctrl"); print TEMP_CTRL "$_"; # contains information for swap.pl close TEMP_CTRL; system("swap.pl"); # swap.pl reads temp_ctrl to know which part of + the dataset to process sleep(1); }

The problem is, that the system command waits until the process is finished, which means there is no parallelization at all. And the problem with exec command is that it does not return.

Can anyone help?

UPDATE: Thanks for the suggestions. I found

system(1,"swap.pl")
to work well.

Comment on exec vs. system
Select or Download Code
Re: exec vs. system
by jethro (Monsignor) on Jul 07, 2011 at 12:29 UTC

    system("swap.pl &") will put swap.pl into the background (a feature of the shell).

    By the way, wouldn't it be easier to give the data as parameter to swap.pl, i.e. system("swap.pl \"$_\" &") ? This would only work for non-binary data naturally and you would have to escape " \ and $ in the string

      Thanks for that reply but system("swap.pl &") seems not to put the process into the background. The master script still waits untill swap.pl is finished.
        Ah, so you probably are on Windows. At least I never heard of system(1,...) being sensible syntax, so probably this is an extension in your windows perl dialect
      By the way, wouldn't it be easier to give the data as parameter to swap.pl, i.e. system("swap.pl \"$_\" &")? This would only work for non-binary data naturally and you would have to escape " \ and $ in the string.
      Not only easier, but more robust. Those of us who have been burned in almost every possible way cringe at the assumption that swap.pl will be done with the control file in less than one second, when it will be replaced by the control file for another instance. If the control data is too complex to pass as an argument, then give each control file a different name and pass the name to swap.pl as an argument.
Re: exec vs. system
by derby (Abbot) on Jul 07, 2011 at 12:32 UTC

    Well ... there are a number of ways to put swap.pl into a background process but you'll find the easiest approach to your problem is to use Parallel::ForkManager and totally avoid the use of an external file and/or an external program.

    -derby
Re: exec vs. system
by sundialsvc4 (Monsignor) on Jul 07, 2011 at 12:34 UTC

    The “ruling constraint” of your program is probably not CPU capacity:   it is I/O.   Throwing additional processes at an I/O-bound algorithm generally makes it dramatically slower, not faster.

      The “ruling constraint” of your program is probably not CPU capacity:

      On what information do you base that conclusion?

Re: exec vs. system
by tokpela (Chaplain) on Jul 07, 2011 at 19:34 UTC

    Based on the above responses (and the fact that you are using Windows), maybe using threads might be a better solution for you?

    See BrowserUK's posts for some good examples. Try this one which uses threads to run system commands or this one which uses Thread::Queue (but no system command).

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://913169]
Approved by BrowserUk
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (13)
As of 2014-04-23 17:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (549 votes), past polls