Update: My next attempt compares running with threads and non-threads on the Mac and Linux. There is something strange about strftime that causes the script to slow down either with threads or non-threads depending on the OS.
Update: The serial code runs faster on a Linux VM. For some reason, the strftime function degrades in performance when running with many workers (even threads on Linux). I'm not sure why.
In my testing, strftime performs poorly when many workers call it simultaneously. This is fine with threads, but must limit the number of workers.
On my laptop (running Mac OS X), the serial code completes in 19.131 seconds for a 500 MB file and MCE completing in 6.569 seconds. Most of that time is coming from strftime. I verified this by replacing $A = strftime with $A = $Y which completes in 1.842 seconds.
use POSIX qw(strftime);
my $infile = $ARGV;
my $outfile = $ARGV;
open(DATAOUT, ">", $outfile);
## Workers process chunks in parallel until completed.
## Output order is preserved via MCE::Candy::out_iter_fh
chunk_size => "2m", max_workers => 4, use_slurpio => 1,
gather => MCE::Candy::out_iter_fh(\*DATAOUT),
use_threads => 1
my ($mce, $chunkRef, $chunkID) = @_;
my ($output, @Fields, $X, $Y, $A, $B, $C, $D) = ("");
open my $CHUNKIN, "<", $chunkRef;
while( my $line = <$CHUNKIN> )
@Fields = split(',', $line, 9);
$X = $Fields;
$Y = substr $X, 0, 10;
$A = strftime "%M,%Y,%m,%d,%H,%j,%W,%u,%A", gmtime $Y;
$B = substr($A, 0, index($A, ','));
$C = int($B/5);
$D = int($B/15);
$output .= $line.",$Y,$A,$C,$D\n";
Kind regareds, Mario.
Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
Read Where should I post X? if you're not absolutely sure you're posting in the right place.
Please read these before you post! —
Posts may use any of the Perl Monks Approved HTML tags:
You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
- a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
Link using PerlMonks shortcuts! What shortcuts can I use for linking?
See Writeup Formatting Tips and other pages linked from there for more info.
| & || & |
| < || < |
| > || > |
| [ || [ |
| ] || ] |