Re^4: Using MCE to write to multiple files.

[BrowserUk]

Damn! That's a complicated way of doing something very simple. It took me forever to work out where $chunkIds originated.

They start life as an integer list wrapped in a anonymous array:

    threads->create( \&JobAdder, $q, $maxJobs, [ 0 .. 99 ] );

And get past individually to AddJob() here:

        AddJob( $q, shift @$inputs );

Where they get pushed individually, but wrapped in individual anonymous arrays, onto the queue here:

    $q->enqueue( [@_] );

Then those anonymous array refs get dequeued here:

    while( defined( my $argRef = $q->dequeue ) ) {

And the individual integer are then unwrapped from them here:

        $callBack->( @$argRef );

Before being passed to where they actually get used to number some files here:

sub GetFibby {
    my( $chunkId ) = @_;
    use autodie qw/ open close /;
    open my( $outfh ), '>', "fibojob-$chunkId.txt";

All of that to achieve the equivalent of:

$q->enqueue( 0 .. 99 );

Here's a simpler version that avoids the obfuscation:

#! perl -slw
use strict;
use threads;
use threads::Q;

sub fibonacci {
    my $n = shift;
    return undef if $n < 0;
    my $f;
    if( $n == 0 ) {
        $f = 0;
    } elsif( $n == 1 ) {
        $f = 1;
    } else {
        $f = fibonacci( $n - 1 ) + fibonacci( $n - 2 );
    }
    return $f;
}

sub worker {
    my $tid = threads->tid;
    my( $Q, $path ) = @_;
    chdir $path or die "$path : $!";

    while( defined( my $chunkId = $Q->dq ) ) {
        open my $out, '>', "fibojob-$chunkId.txt" or die "$chunkId : $
+!";
        print $out join "\t", map fibonacci( $_ ), 1 .. 10;
        close $out;
    }
}

our $T //= 8;
our $P //= '.';
our $M //= 99;

## The pattern is simple

my $Q = threads::Q->new( $T*2 );                                 ## Cr
+eate a queue.
my @threads = map threads->create( \&worker, $Q, $P ), 1 .. $T;  ## Cr
+eate some workers to read from that queue.
$Q->nq( 0 .. $M );                                               ## Qu
+eue some work.
$Q->nq( ( undef ) x $T );                                        ## Te
+ll the workers they are done.
$_->join for @threads;                                           ## An
+d wait for them to finish.