http://www.perlmonks.org?node_id=1165734

Paul.Unix has asked for the wisdom of the Perl Monks concerning the following question:

I have a multi threaded Perl script (master-workers architecture) in which the workers call an external application and need to process the standard output and standard error. I tried Capture::Tiny but that crashed the worker threads if a lot of workers were busy processing. Now, I use IPC::Run3 but this also fails, although less often as Capture::Tiny. Is there a safe way to call external applications from parallel running worker threads and being able to process the standard output and standard error? The idea is that the external calls can run in parallel so semaphore locking seems not the way to go to me. My current run3 call
my @spaziooptions = ($spaziocmd, "-m" , $transref->{'QMANAGER'} ,"-q" +, $destination_queue, "-i" , $transref->{'POLLERFILE'} ,"-d" , $desti +nation_filename , "-k" , $transref->{'DATATYPE'}, "-s", $global_hostn +ame, "-c", $cor_id); my $cmdout; my $cmderr; run3 \@spaziooptions, \undef, \$cmdout, \$cmderr; my $cmdexit = $?;

Replies are listed 'Best First'.
Re: thread save calling an external command
by BrowserUk (Patriarch) on Jun 15, 2016 at 15:13 UTC
    need to process the standard output and standard error.

    Do you need them as separate streams? How are you processing them? Could you show the code where you are processing the output?


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice. Not understood.
      The idea is to have parallel processes running to avoid long running processes block processing other items. I am mainly interested in the error output of the external commands. The exit codes of the external command are not suitable to tell what went wrong.

        If you want both stdout and stderr (and don't mind them mixed together:

        #! perl -slw use strict; use threads; sub thread { my $output = ''; open CMD, '-|', q[ perl -E"say( qq[$$:saying $_] ), warn( qq[$$:wa +rning $_\n] ), sleep(1) for 1 .. 4" 2>&1 ] or die $!; $output .= $_ while <CMD>; close CMD; return $output; } printf "Got\n'%s'\n", $_->join for map threads->create( \&thread ), 1 +.. 4; __END__ C:\test>t-junk Got '33168:warning 1 33168:warning 2 33168:warning 3 33168:warning 4 33168:saying 1 33168:saying 2 33168:saying 3 33168:saying 4 ' Got '30208:warning 1 30208:warning 2 30208:warning 3 30208:warning 4 30208:saying 1 30208:saying 2 30208:saying 3 30208:saying 4 ' Got '15844:warning 1 15844:warning 2 15844:warning 3 15844:warning 4 15844:saying 1 15844:saying 2 15844:saying 3 15844:saying 4 ' Got '15032:warning 1 15032:warning 2 15032:warning 3 15032:warning 4 15032:saying 1 15032:saying 2 15032:saying 3 15032:saying 4 '

        If you only want stderr:

        #! perl -slw use strict; use threads; sub thread { my $output = ''; open CMD, '-|', q[ perl -E"say( qq[$$:saying $_] ), warn( qq[$$:wa +rning $_\n] ), sleep(1) for 1 .. 4" 2>&1 >nul ] or die $!; $output .= $_ while <CMD>; close CMD; return $output; } printf "Got\n'%s'\n", $_->join for map threads->create( \&thread ), 1 +.. 4; __END__ C:\test>t-junk Got '34456:warning 1 34456:warning 2 34456:warning 3 34456:warning 4 ' Got '32700:warning 1 32700:warning 2 32700:warning 3 32700:warning 4 ' Got '19172:warning 1 19172:warning 2 19172:warning 3 19172:warning 4 ' Got '22176:warning 1 22176:warning 2 22176:warning 3 22176:warning 4 '

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice. Not understood.
Re: thread save calling an external command
by talexb (Chancellor) on Jun 15, 2016 at 15:45 UTC

    Can you be more specific about how IPC::Run3 fails? It's difficult to solve the problem when we don't even know what the problem is. OK -- almost impossible.

    Alex / talexb / Toronto

    Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

Re: thread save calling an external command
by oiskuu (Hermit) on Jun 15, 2016 at 17:34 UTC

    I don't think run3 can be used in a threaded environment. Looking at the module code, it makes the redirects, then does system() i.e. fork/exec. A thread-safe version would open the streams; fork(); redirect i/o; exec().

    The second problem is waiting upon the processes. You'll need a mechanism to catch child notifications and forward them to the interested threads (via Thread::Queue probably).

    In the end, it might be smarter to implement a separate job-controller thread to launch and reap the external jobs.

    Update. On a second thought, if your workers only ever wait on a single process at a time, you can get away with just the waitpid($pid, 0); and no job control is necessary...

Re: thread save calling an external command
by salva (Canon) on Jul 06, 2016 at 09:26 UTC
    There is a very simple solution to that problem: Redirect the output of every child process to temporary file and once it terminates, read it back for processing. That way you only have to take care of concurrently launching and reaping processes, forgetting about the IO.

    It may seem pretty inefficient forcing the data through the file-system, but in practice, usually it is not unless the slave processes write huge quantities of data.

      Simple... and neat using File::Temp:

      use File::Temp; sub cmdrun { my $cmd = shift; my $out = File::Temp->new(); my $err = File::Temp->new(); local $/; `$cmd >$out 2>$err`; $?>>8, scalar <$out>, scalar <$err> } use Data::Dumper; print Dumper cmdrun( q(perl -le "warn q(NOISE); print q(OUTPUT); exit +42;") );