Re: PerlIO::gzip and Parallel::ForkManager do not play nice

[marioroy]

Hi Pascal666,

The following are three parallel demonstrations using MCE::Flow, MCE::Hobo, and threads. The shared input and output handles are managed by MCE::Shared.

MCE::Flow and MCE::Shared

Many-Core Engine provides chunking abilities not used here.

use strict;
use warnings;

use MCE::Flow;
use MCE::Shared;
use PerlIO::gzip;

my $IN  = MCE::Shared->handle( '<:gzip', 'wat.paths.gz' );
my $OUT = MCE::Shared->handle( '>', \*STDOUT );

mce_flow { max_workers => 12 }, sub {
   while (my $file = <$IN>) {
      print $OUT length($file).":$file" if length($file) > 142;
   }
};

close $IN;

MCE::Hobo and MCE::Shared

A Hobo is a migratory worker inside the machine that carries the asynchronous gene. Hobos are equipped with threads-like capability for running code asynchronously. Unlike threads, each hobo is a unique process to the underlying OS. The IPC is managed by MCE::Shared, which runs on all the major platforms including Cygwin.

use strict;
use warnings;

use MCE::Hobo;
use MCE::Shared;
use PerlIO::gzip;

my $IN  = MCE::Shared->handle( '<:gzip', 'wat.paths.gz' );
my $OUT = MCE::Shared->handle( '>', \*STDOUT );

sub task {
   while (my $file = <$IN>) {
      print $OUT length($file).":$file" if length($file) > 142;
   }
}

MCE::Hobo->create('task') for 1 .. 12;

# do other stuff if desired

$_->join for MCE::Hobo->list;

close $IN;

threads and MCE::Shared

The code for MCE::Hobo and threads are very similar.

use strict;
use warnings;

use threads;
use MCE::Shared;
use PerlIO::gzip;

my $IN  = MCE::Shared->handle( '<:gzip', 'wat.paths.gz' );
my $OUT = MCE::Shared->handle( '>', \*STDOUT );

sub task {
   while (my $file = <$IN>) {
      print $OUT length($file).":$file" if length($file) > 142;
   }
}

threads->create('task') for 1 .. 12;

# do other stuff if desired

$_->join for threads->list;

close $IN;

All three examples work across the board including with Perl on Windows and Cygwin, provided the OS has the gzip binary and the PerlIO::gzip module installed.

Update:

The upcoming MCE::Shared 1.002 release will support the following construction by allowing the main or worker process to handle the error. I've been wanting for the shared open call to feel like the native open call.

use MCE::Shared 1.002;

mce_open my $IN, "<:gzip", "wat.paths.gz" or die "open error: $!";
mce_open my $OUT, ">", \*STDOUT or die "open error: $!";

Kind regards, Mario