Re: Speeding up stalled script

[marioroy]

The following is another way. The block receives an array reference. Between the two, slurping is faster 2.023s versus 2.904s for a FASTA file containing 13.7 million lines excluding headers.

use MCE::Flow;

my $input1 = shift @ARGV;
my @lengths;

mce_flow_f {
   chunk_size => '2m', max_workers => 'auto',
   gather => \@lengths
},
sub {
   my ($mce, $chunk_ref, $chunk_id) = @_;
   my (@chunk_lengths);

   chomp @{ $chunk_ref };
   foreach (@{ $chunk_ref }) {
      push @chunk_lengths, length $_ unless />/;
   }

   MCE->gather(@chunk_lengths);

}, $input1;

MCE::Flow::finish;

print scalar @lengths, "\n";