in reply to Re: About text file parsing
in thread About text file parsing
Hi again,
One may want to have the manager-process receive and loop through @sample and @good. That will incur an additional CPU core for the manager-process itself.
use strict; use warnings; use MCE; open my $sample_fh, ">", "sample.txt" or die "open error: $!"; open my $good_fh, ">", "good.txt" or die "open error: $!"; # worker function sub task { my ( $mce, $slurp_ref, $chunk_id ) = @_; my ( @sample, @good ); # open file handle to scalar ref open my $input_fh, "<", $slurp_ref; # append to scalars inside the loop while (<$input_fh>) { if (/^sample\s+(\S+)/) { push @sample, $1; } elsif (/^good\s+(\S+)/) { push @good, $1; } } close $input_fh; # send arrays to the manager-process MCE->gather(\@sample, \@good); } # manager function sub gather { my ( $sample, $good ) = @_; # process sample for ( @{ $sample } ) { ; } # process good for ( @{ $good } ) { ; } } # spawn workers early, optionally my $mce = MCE->new( chunk_size => '1m', # 1 megabyte max_workers => 4, use_slurpio => 1, user_func => \&task, gather => \&gather, )->spawn; # process input file(s) $mce->process({ input_data => "test.txt" }); # shutdown workers $mce->shutdown; # close output handles close $sample_fh; close $good_fh;
The extra time comes from workers appending to local arrays. Likewise, the manager-process receiving and looping through the arrays. There are 4 workers and the manager process running simultaneously on a machine with 4 real cores.
$ time perl test_demo.pl real 0m9.932s user 0m43.956s sys 0m0.452s
Update:
Interestingly, Perl v5.20 and higher take 2x longer to run. I'm not sure why. Yikes, possibly from regular expression? This is on my TODO list to check why. The above was captured from Perl v5.18.2 on the same machine.
$ time /opt/perl-5.20.3/bin/perl test_demo.pl real 0m20.858s user 1m20.164s sys 0m8.488s
Regards, Mario
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^3: About text file parsing
by marioroy (Prior) on Aug 30, 2018 at 14:17 UTC |
In Section
Seekers of Perl Wisdom