Re: Fast provider feeding slow consumer

[NetWallah]

I belive BrowserUk originally wrote this code - I could not find his original post, so I'm sharing again.

This uses threads/queues, and provides the functionality you are seeking, aside from being an excellent reference implementation:

#! perl -slw
use strict;
use threads;
use Thread::Queue;

$| = 1; #$OUTPUT_AUTOFLUSH

our $KIDS ||= 10;
our $WORK ||= 500;
our $SLEEP||= 5;

sub kid {
    my( $Q ) = shift;
    my $tid = threads->tid;
    my $count=0;
    printf "Kid: %02d started\n", $tid;

    ## Pick a work item of the queue and process it
    while( my $work = $Q->dequeue ) {
        printf "Kid: %02d processing work item '%s'\n", $tid, $work;
        $count ++;

        ## Replace the sleep with the code to process teh work items
        rand > 0.7 and sleep rand( $SLEEP );
    }
    print "kid: $tid ending after processing $count items.\n";
}

## A queue for communications
my $Q = new Thread::Queue;

## Start the kids
my @kids = map{ threads->create( \&kid, $Q ) } 1 .. $KIDS;

## Wait till they're all up and running
sleep 1 until @{[ threads->list ]} == $KIDS;

## Feed the queue with work
## The limit just ensure we don't fill lots of memory
for my $workitem ( 1 .. $WORK ) {
    sleep 1 while $Q->pending > $KIDS *10;
    print "Queueing work item $workitem";
    $Q->enqueue( $workitem );
}

## Tell them to stop
$Q->enqueue( (undef) x $KIDS );

## And wait for them to do so.
$_->join for @kids;

[leostereo]

Excellent note my friend, I will study it.