Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Using Parallel::ForkManager on multiple files using backtick operators for multiple files being processed simulnaneously

by 1nickt (Abbot)
on Feb 19, 2020 at 19:18 UTC ( #11113172=note: print w/replies, xml ) Need Help??


in reply to Using Parallel::ForkManager on multiple files using backtick operators for multiple files being processed simulnaneously

Hi, you didn't supply sample input or output, or describe what the external program does, so the following is untested, but here's something to try using MCE.

#!/usr/bin/env perl -w use strict; use warnings; use MCE::Loop; use Capture::Tiny 'capture_stdout'; use Text::CSV 'csv'; use Try::Tiny; use JSON; MCE::Loop::init { max_workers => 8, use_slurpio => 1 }; my $data_filename = 'all.txt'; my $fail_filename = 'failed.tsv'; my $cmd = 'cfn_nag_scan'; my @args = qw/ -o json --input-path /; my @results = mce_loop_f { my ($mce, $slurp_ref, $chunk_id) = @_; my @failures; while ( $$slurp_ref =~ /([^\n]+\n)/mg ) { my $line = $1; my $json = try { return decode_json(capture_stdout { system($cmd, @args, $ +line) }); } catch { warn "processing $line failed: $_"; return; }; next unless $json; my $failure_count = $json->{file_results}{failure_count}; push @failures, [ $json->{filename} => $failure_count ] if $fa +ilure_count; } # Gather results MCE->gather(@failures); } $data_filename; unshift @results, [qw/ Filename Failures /]; csv (in => \@results, out => $fail_filename, sep_char=> "\t"); __END__

update: added json decoding and exception handling; changed return to next (thx marioroy)

Hope this helps!


The way forward always starts with a minimal test.
  • Comment on Re: Using Parallel::ForkManager on multiple files using backtick operators for multiple files being processed simulnaneously
  • Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11113172]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (9)
As of 2020-04-04 09:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    The most amusing oxymoron is:
















    Results (32 votes). Check out past polls.

    Notices?