Re: Threads memory consumption

[Random_Walk]

something along these lines may get you started...

use strict;
use warnings;
use threads;
use Thread::Queue;

my $wq = Thread::Queue->new();    # Build a queue where you put work

my $thread_limit = 4;     # how many threads do you want?

my @thr = map {  # create threads and store the handles in @thr
    threads->create(
        sub {
            while (my $item = $wq->dequeue()) { # finish when undef
                # Do something with item
            }
        } );
} 1..$thread_limit;

while (I get $data to process) {
    $wq->enqueue($data);
}

# Tell the threads we are done
$wq->enqueue(undef) for @thr;

# Wait for all the threads to finish
$_->join() for @thr;

Now go and read the docco for Threads and Thread::Queue. Its pretty good.

Cheers,
R.

[mojo2405]

thanks alot ! do you think it will solve my memory consumption problem ?

[Random_Walk]

Without seeing more of your code its impossible to tell.

I have seen multi threaded code eat all the memory when a large shared hash is first built containing all the work to be done, and the threads are then created. In this way each thread gets a copy of the hash, and your memory usage is terrible. Using a queue you use a lot less memory. If however you have some other code in there leaking memory, then changing the way you call threads is unlikely to solve it.

Are you using strict and warnings? Do you declare all your variables in the smallest possible scope? Can you create a small test version of your code that shows this memory leaking behaviour?

Cheers,
R.