Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^2: PerlIO file handle dup

by marioroy (Priest)
on Mar 07, 2017 at 09:27 UTC ( #1183824=note: print w/replies, xml ) Need Help??


in reply to Re: PerlIO file handle dup
in thread PerlIO file handle dup

Greetings,

To decrease the number of trips to and from the shared-manager, one can provide a suffix (k * 1024) or (m * 1024 * 1024) for the 3rd argument to read. That there enables chunk IO. Not to worry, the shared-manager completes reading until reaching the end of line or record. Notice $. It is the chunk_id, not the actual line number. The chunk_id value is important when output order is desired.

OP's script involving semaphore + yield: 3.6 seconds. Shared handle (non-chunking): 1.1 seconds.

Below, chunking completes in 0.240 seconds which is the total running time including initial gzip.

use strict; use threads; use MCE::Shared; { open my $fh, '|-', 'gzip > test.txt.gz'; foreach (1..100000) { print {$fh} sprintf('%04d',$_).('abc123' x 10)."\n"; } close $fh; } { mce_open my $fh, '-|', 'gzip -cd test.txt.gz' or die "open error: +$!\n"; mce_open my $out, '>', \*STDOUT or die "open error: $!\n"; my @thrs; foreach (1..3) { push @thrs, threads->create('test'); } $_->join() foreach @thrs; close($fh); sub test { my $tid = threads->tid(); # using shared output to not garble among threads while (1) { my $n_chars = read $fh, my($buf), '4k'; last if (!defined $n_chars || $n_chars <= 0); print {$out} "## thread: $tid, chunkid: $.\n".$buf; } } }

Regards, Mario.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1183824]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (4)
As of 2019-06-17 23:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Is there a future for codeless software?



    Results (80 votes). Check out past polls.

    Notices?
    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!