Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

@array_commands to execute simultaneously and collect output on single log

by nicopelle (Novice)
on Dec 15, 2016 at 13:43 UTC ( #1177838=perlquestion: print w/replies, xml ) Need Help??
nicopelle has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I've got an array with, more or less, 80 system commands (all different one each other) ready to execute on AIX lpar.
@array_commands=qw( command1 command2 command3 .. .. command79 command80 );
Is there a smart way to launch simultaneously(this is mandatory) and collecting all output in a single log file ? Thanks for your time, Nicola.
  • Comment on @array_commands to execute simultaneously and collect output on single log
  • Download Code

Replies are listed 'Best First'.
Re: @array_commands to execute simultaneously and collect output on single log
by choroba (Bishop) on Dec 15, 2016 at 14:47 UTC
    I'm not sure about AIX, but the following worked for me on Linux, and it took less than 6 seconds:
    #!/usr/bin/perl use warnings; use strict; my @commands = ( 'sleep 3 && find /usr/share/doc', 'sleep 4 && find /opt', 'sleep 5 && df' ); my @pids; for my $i (0 .. $#commands) { if (my $pid = fork) { push @pids, $pid; } elsif (defined $pid) { 0 == system "$commands[$i] > $i.o" or die "Can't run $commands +[$i]"; exit } else { die "Can't fork"; } } for my $i (0 .. $#commands) { print "=== $commands[$i] ===\n"; waitpid $pids[$i], 0; open my $fh, '<', "$i.o" or die "$i.o: $!"; print while <$fh>; unlink "$i.o"; }

    The output from the first command goes first, etc.

    Update

    Or, without temporary files:

    #!/usr/bin/perl use warnings; use strict; my @commands = ( 'sleep 3 && find /usr/share/doc', 'sleep 4 && find /opt', 'sleep 5 && df' ); my @fhs; for my $i (0 .. $#commands) { open $fhs[$i], '-|', $commands[$i] or die "Can't run $commands[$i] +"; } for my $i (0 .. $#commands) { print "=== $commands[$i] ===\n"; print while readline $fhs[$i]; }

    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
      Thanks !
Re: @array_commands to execute simultaneously and collect output on single log
by BrowserUk (Pope) on Dec 15, 2016 at 19:43 UTC

    This will start four processes on my 4 core system within 0.2 of a second of each other:

    #! perl -sw use strict; use threads; use threads::shared; use Thread::Queue; use Time::HiRes qw[ time ]; our $T //= 4; ## 80 my $sem :shared = 0; my @commands = map qq[perl -MTime::HiRes=time -le"print \$\$, ':', tim +e()"], 1 .. $T; sub doit { my( $cmd, $Q ) = @_; lock $sem; my $pid = open my $pipe, '-|', $cmd or die $!; $Q->enqueue( "$pid:$_" ) while <$pipe>; $Q->enqueue( undef ); } my $Q = new Thread::Queue; { lock $sem; async( \&doit, $_, $Q )->detach for @commands; sleep 2; } open OUT, '>', 'log' or die $!; for( 1 .. $T ) { print OUT while defined( $_ = $Q->dequeue ); } close OUT; __END__ c:\test>type log 25692:25692:1481831121.12942 26080:26080:1481831121.18679 26088:26088:1481831121.2571 25704:25704:1481831121.32448

    To get closer than that you'd need to move the semaphoring into the child processes.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: @array_commands to execute simultaneously and collect output on single log
by BrowserUk (Pope) on Dec 15, 2016 at 14:26 UTC
    simultaneously(this is mandatory)

    Do you mean:

    • Start them all running one after the other without waiting for the preceding one to finish?
    • Or: Have them all start running at the exact same moment in time?

    And why? IE. What drives the need for simultaneous running?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
    In the absence of evidence, opinion is indistinguishable from prejudice.
      of course the second one: "Have them all start running at the exact same moment in time"

        So, you are running on 80+ core machines then?

Re: @array_commands to execute simultaneously and collect output on single log
by tybalt89 (Priest) on Dec 15, 2016 at 21:20 UTC
    #!/usr/bin/perl # http://perlmonks.org/?node_id=1177838 use strict; use warnings; my @cmds = ('date +%S.%N') x 80; my @fhs; open $fhs[@fhs], '-|', $_ or die "$! on open of $_" for @cmds; print map <$_>, @fhs;

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1177838]
Approved by Corion
Front-paged by trippledubs
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (2)
As of 2018-08-18 02:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Asked to put a square peg in a round hole, I would:









    Results (184 votes). Check out past polls.

    Notices?