Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re^2: PerlIO file handle dup

by marioroy (Vicar)
on Mar 07, 2017 at 09:27 UTC ( #1183824=note: print w/replies, xml ) Need Help??

in reply to Re: PerlIO file handle dup
in thread PerlIO file handle dup


To decrease the number of trips to and from the shared-manager, one can provide a suffix (k * 1024) or (m * 1024 * 1024) for the 3rd argument to read. That there enables chunk IO. Not to worry, the shared-manager completes reading until reaching the end of line or record. Notice $. It is the chunk_id, not the actual line number. The chunk_id value is important when output order is desired.

OP's script involving semaphore + yield: 3.6 seconds. Shared handle (non-chunking): 1.1 seconds.

Below, chunking completes in 0.240 seconds which is the total running time including initial gzip.

use strict; use threads; use MCE::Shared; { open my $fh, '|-', 'gzip > test.txt.gz'; foreach (1..100000) { print {$fh} sprintf('%04d',$_).('abc123' x 10)."\n"; } close $fh; } { mce_open my $fh, '-|', 'gzip -cd test.txt.gz' or die "open error: +$!\n"; mce_open my $out, '>', \*STDOUT or die "open error: $!\n"; my @thrs; foreach (1..3) { push @thrs, threads->create('test'); } $_->join() foreach @thrs; close($fh); sub test { my $tid = threads->tid(); # using shared output to not garble among threads while (1) { my $n_chars = read $fh, my($buf), '4k'; last if (!defined $n_chars || $n_chars <= 0); print {$out} "## thread: $tid, chunkid: $.\n".$buf; } } }

Regards, Mario.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1183824]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2020-02-29 14:40 GMT
Find Nodes?
    Voting Booth?
    What numbers are you going to focus on primarily in 2020?

    Results (128 votes). Check out past polls.