in reply to PerlIO file handle dup


Welcome to the world of multi-threads or cores for that matter. It's one thing to make possible on the UNIX platform and altogether something else on the Windows platform. And not to forget Cygwin.

For me, Perl is a box of crayons. MCE and MCE::Shared are my paintings. I imagine folks around the world joined together in a relay race. I have no idea what we're aiming for. It just happens to be my turn at this moment in time. The tools given me are a box of crayons named Perl, a laptop, and an agent name Perseverance. Perseverance brought along a long-time friend named Grace. Grace invited Randomness, an Uncle.

The thing is that MCE and MCE::Shared may not be perfect. They are paintings, after all. Paintings take time to paint.

In regards to any slowness, IMHO, let the script fly. Oh, please do. For this use-case, a semaphore is not necessary. Nor is yield. Upon starting a thread, that thread will begin interacting with the shared-manager immediately. That is why the same thread ID is shown repeatedly for many lines in the output. Eventually, the 2nd thread has completed spawning and joins the 1st thread. Thread 3 joins later due to being spawned last.

I up'ed the count to 100k. The OP's script with semaphore + yield takes 3.6 seconds to run on a laptop (2.6 GHz - Core i7 Haswell). Removing the semaphore + yield allows the script to complete in 1.1 seconds. The latter includes threads writing to a shared output handle. In case it was missed, I removed the line to autoflush STDOUT; e.g. $| = 1. There's no reason to slow down IO. Let Perl fly. Ditto for MCE::Shared and workers.

use strict; use threads; use MCE::Shared; { open my $fh, '|-', 'gzip > test.txt.gz'; foreach (1..100000) { print {$fh} sprintf('%04d',$_).('abc123' x 10)."\n"; } close $fh; } { mce_open my $fh, '-|', 'gzip -cd test.txt.gz' or die "open error: +$!\n"; mce_open my $out, '>', \*STDOUT or die "open error: $!\n"; my @thrs; foreach (1..3) { push @thrs, threads->create('test'); } $_->join() foreach @thrs; close($fh); sub test { my $tid = threads->tid(); # using shared output to not garble among threads while ( my $line = <$fh> ) { print {$out} "thread: $tid, line: $., ".$line; } } }

It can run faster. To be continued in the next post.

Regards, Mario.