comment on

Hmm, why don't make it simple, at least if you're on Linux?

# We have some jobs which take some time...
@Cmds = ('curl "http://www.perlmonks.org/?node=Newest%20Nodes"',
         'fsck -n /',
         './calculate_high_accurate_pi');
# Let's start them...
my @PIDs;
my $Tempfile = '/tmp/job_controller.'.$$;
for (0..$#Cmds) {
 $PIDs[$_] = `$Cmds[$_] >$Tempfile.$_ 2>&1 & echo $!`;
}
# Ok, they all are running now
while (kill 0,@PIDs) {
 sleep 1;
}
for (0..$#Cmds) {
 open my $TmpFH,$Tempfile.'.'.$_;
 print "----- $Cmds[$_] returned: -----\n".
       join('',<$TmpFH>);
 close $TmpFH;
 unlink $Tempfile.'.'.$_;
}
print "Everthing done!\n";
[download]

I defined three jobs: One waits for the network, one for the disk and one for the CPU.
We need two helpers: One array for storing the PIDs of the running jobs and one prefix for temprary files we're going to create.
Each job is started with the help of bash: The & brings the main job into background and now $! contains the new PID of the job. (If your jobs finish their output with some kind of end_of_file marker (like </html>), you could check for this later, but beware of crashed jobs - you'll wait forever!) Each job writes his STDOUT and STDERR into a combined file which is called like our defined prefix and the job number.
Now they're running, your script could do something useful here, but we're just going to wait for them. kill sends the signal 0 (which is some kind of noop, it doesn't actually reach the process) to all our processes and returns the count of processes which exist. It would loop as long as there is at least one of our childs running.
As soon as we know that the job is finished, we could just read the output and do whatever we like to to.

Improvements/Comments

Look at the time your jobs need to finish. If they take approx 5 seconds each, don't sleep for a second,

select undef,undef,undef,.5;
[download]

would do the job. If they run for hours, you may want to sleep 30 because half a minute on top doesn't care.

If you work with the output or start follow-up jobs as soon as the output from the first job is complete, kill 0,$PID them one-by-one and as soon as one has finished, start the post-processing. The results from the others won't be lost - you'ld find them as soon as you're done with the first processing.

You could even catch the outputs while they're running (don't forget to add $|=1 if your jobs are perl-scripts). Think of a webpage with 3 progress bars. Every one or two seconds, you read the last lines/bytes from the output file and update the process bars for each job.

As you may have noticed, I usually prefer the simple way instead of throwing a dozen modules against a simple problem.

In reply to Re: running multiple system commands in parallel by Sewi
in thread running multiple system commands in parallel by incognito129

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Perl Monk, Perl Meditation
	PerlMonks