thread save calling an external command

[Paul.Unix]

I have a multi threaded Perl script (master-workers architecture) in which the workers call an external application and need to process the standard output and standard error. I tried Capture::Tiny but that crashed the worker threads if a lot of workers were busy processing. Now, I use IPC::Run3 but this also fails, although less often as Capture::Tiny. Is there a safe way to call external applications from parallel running worker threads and being able to process the standard output and standard error? The idea is that the external calls can run in parallel so semaphore locking seems not the way to go to me. My current run3 call

my @spaziooptions = ($spaziocmd, "-m" , $transref->{'QMANAGER'} ,"-q" 
+, $destination_queue, "-i" , $transref->{'POLLERFILE'} ,"-d" , $desti
+nation_filename , "-k" , $transref->{'DATATYPE'}, "-s", $global_hostn
+ame, "-c", $cor_id);
    my $cmdout;
    my $cmderr;
    run3 \@spaziooptions, \undef, \$cmdout, \$cmderr;
    my $cmdexit = $?;

[BrowserUk]

need to process the standard output and standard error.

Do you need them as separate streams? How are you processing them? Could you show the code where you are processing the output?

[Paul.Unix]

The idea is to have parallel processes running to avoid long running processes block processing other items. I am mainly interested in the error output of the external commands. The exit codes of the external command are not suitable to tell what went wrong.

[BrowserUk]

If you want both stdout and stderr (and don't mind them mixed together:

#! perl -slw
use strict;
use threads;

sub thread {
    my $output = '';
    open CMD, '-|', q[ perl -E"say( qq[$$:saying $_] ), warn( qq[$$:wa
+rning $_\n] ), sleep(1) for 1 .. 4" 2>&1 ] or die $!;
    $output .= $_ while <CMD>;
    close CMD;
    return $output;
}

printf "Got\n'%s'\n", $_->join for map threads->create( \&thread ), 1 
+.. 4;

__END__
C:\test>t-junk
Got
'33168:warning 1
33168:warning 2
33168:warning 3
33168:warning 4
33168:saying 1
33168:saying 2
33168:saying 3
33168:saying 4
'
Got
'30208:warning 1
30208:warning 2
30208:warning 3
30208:warning 4
30208:saying 1
30208:saying 2
30208:saying 3
30208:saying 4
'
Got
'15844:warning 1
15844:warning 2
15844:warning 3
15844:warning 4
15844:saying 1
15844:saying 2
15844:saying 3
15844:saying 4
'
Got
'15032:warning 1
15032:warning 2
15032:warning 3
15032:warning 4
15032:saying 1
15032:saying 2
15032:saying 3
15032:saying 4
'

If you only want stderr:

#! perl -slw
use strict;
use threads;

sub thread {
    my $output = '';
    open CMD, '-|', q[ perl -E"say( qq[$$:saying $_] ), warn( qq[$$:wa
+rning $_\n] ), sleep(1) for 1 .. 4" 2>&1 >nul ] or die $!;
    $output .= $_ while <CMD>;
    close CMD;
    return $output;
}

printf "Got\n'%s'\n", $_->join for map threads->create( \&thread ), 1 
+.. 4;

__END__
C:\test>t-junk
Got
'34456:warning 1
34456:warning 2
34456:warning 3
34456:warning 4
'
Got
'32700:warning 1
32700:warning 2
32700:warning 3
32700:warning 4
'
Got
'19172:warning 1
19172:warning 2
19172:warning 3
19172:warning 4
'
Got
'22176:warning 1
22176:warning 2
22176:warning 3
22176:warning 4
'

[talexb]

Now, I use IPC::Run3 but this also fails, although less often as Capture::Tiny ..

Can you be more specific about how IPC::Run3 fails? It's difficult to solve the problem when we don't even know what the problem is. OK -- almost impossible.

[oiskuu]

I don't think run3 can be used in a threaded environment. Looking at the module code, it makes the redirects, then does system() i.e. fork/exec. A thread-safe version would open the streams; fork(); redirect i/o; exec().

The second problem is waiting upon the processes. You'll need a mechanism to catch child notifications and forward them to the interested threads (via Thread::Queue probably).

In the end, it might be smarter to implement a separate job-controller thread to launch and reap the external jobs.

Update. On a second thought, if your workers only ever wait on a single process at a time, you can get away with just the waitpid($pid, 0); and no job control is necessary...

[salva]

There is a very simple solution to that problem: Redirect the output of every child process to temporary file and once it terminates, read it back for processing. That way you only have to take care of concurrently launching and reaping processes, forgetting about the IO.

It may seem pretty inefficient forcing the data through the file-system, but in practice, usually it is not unless the slave processes write huge quantities of data.

[Anonymous Monk]

Simple... and neat using File::Temp:

use File::Temp;

sub cmdrun {
    my $cmd = shift;
    my $out = File::Temp->new();
    my $err = File::Temp->new();
    local $/;

    `$cmd >$out 2>$err`;
    $?>>8, scalar <$out>, scalar <$err>
}

use Data::Dumper;
print Dumper cmdrun( q(perl -le "warn q(NOISE); print q(OUTPUT); exit 
+42;") );