does threads (still) memleak?

[faxm0dem]

Hi, I'm having huge memleaks using threads; I came upon node_id=625622, which has no conclusion. Here's the thing: running the following code uses up more and more memory, but quite unpredictably depending on the contents of $sub1:

use threads;
use strict;
use warnings;

for (;;) {
 my $thr = threads->new($sub1)->join;
 undef $thr;
}

I'm monitoring rss usage on linux and perl 5.10 using:

i=0; while :; do A=$(ps -C threads3 -o ppid,pid,cmd,rss --no-headers);
+ [ -n "$A" ] && echo $i $A || i=0 ; sleep 1; i=$(($i+1)); done

Here's the output of this, with different values of $sub1:

# $sub1 = sub { my $a=[ 1 .. 10000] };
1 19959 1137 /usr/bin/perl ./threads3 5276
2 19959 1137 /usr/bin/perl ./threads3 5140
3 19959 1137 /usr/bin/perl ./threads3 5700
[...]
642 19959 1137 /usr/bin/perl ./threads3 124244
643 19959 1137 /usr/bin/perl ./threads3 124736
644 19959 1137 /usr/bin/perl ./threads3 125380
# $sub1 = sub { my $a=[ 1 .. 100000] };
1 19959 8329 /usr/bin/perl ./threads3 24176
2 19959 8329 /usr/bin/perl ./threads3 24176
3 19959 8329 /usr/bin/perl ./threads3 24176
[...]
642 19959 8329 /usr/bin/perl ./threads3 28368
643 19959 8329 /usr/bin/perl ./threads3 26036
644 19959 8329 /usr/bin/perl ./threads3 34460
# $sub1 = sub { };
1 19959 13288 /usr/bin/perl ./threads3 3152
2 19959 13288 /usr/bin/perl ./threads3 3796
3 19959 13288 /usr/bin/perl ./threads3 4564
[...]
642 19959 13288 /usr/bin/perl ./threads3 400640
643 19959 13288 /usr/bin/perl ./threads3 401312
644 19959 13288 /usr/bin/perl ./threads3 401980

Any idea what's happening here? $sub1 = sub { my $a= 1 .. 100000 } seems to have minor memleak, while an empty sub does the most damage... I tried using 'forks' instead, which seems to solve the problem. PS: The code proposed by BrowserUk in 625622 also leaks

[BrowserUk]

This is either platform specific, or a bug introduced since 5.8.6 because your code above (with the addition of a counter to display how many threads have been created and joined), doesn't leak on my system using 5.8.6 and threads 1.71:

use threads;
use strict;
use warnings;

print $], ' ', $threads::VERSION, "\n";

my $sub1 = sub { my $a=[ 1 .. 10000] };

my $m = 0;

for (;;) {
    printf "\r%d\t", $m++;
    my $thr = threads->new( $sub1 )->join;
    undef $thr;
}



__END__
c:\test>junk
5.008006 1.71
3077    Terminating on signal SIGINT(2)

For proof, see this image. It shows the cpu, memory usage and IO activity for the process while the code above created and destroyed those 3000+ threads. The memory usage wobbles up and down a bit, but basically stays level at around 2.7 MB.

I note also that in the same thread that you cited, that dave_the_m says that "the bug is fixed in bleed perl"?. (Unfortunately he didn't see fit to inform us of the cause, or the affected platforms:()

So, you could try either reverting to 5.8.6 or possibly upgrading to recently announced the 5.8.9 release candidate.

I tried using 'forks' instead, which seems to solve the problem.

That's a pretty meaningless statement. It's like saying that taking the train cured your car's fuel consumption problem :)

[weismat]

I tried BrowserUk code on my Linux machine and 5.10 and saw no leak as well.
I am using ActiveState build 1001 and CentOS release 4.5.

[Anonymous Monk]

I see the same as you with v5.8.7, ActivePerl Build 813, WinXP.

[BrowserUk]

Thanks. It's looking more and more like the suspicions I've had and voiced about 5.8.8 and threads are right. Something pretty fundemental broke in that build and has only recently been corrected.

That why I stayed back at 5.8.6. (I forgot why I rejected 5.8.7--I seem to recall having some problem with it when it was first released--the installer or maybe PPM didn't work right).

[zentara]

UPDATE Also just tested this on the latest Ubuntu8.10, with Perl5.10 and latest threads.....same memory gain.

I think you should look at OS memory reclamation with threads on linux. Depending on your kernel, OS, computer speed, available free memory, etc, memory reclamation will change. I would guess that with so much demand from the script, and with alot of free memory, the kernel may be playing it safe and not freeing all the time. You might try running it with a sleep 1 in your loop, and give the kernel time to free things up. I'm trying it now, bit it may take a while :-)

I have perl 5.8.8, and the latest threads from cpan, running on linux. When the script below starts, its at 15M, and at count 10,000, it is up to 20M. On the way up it seems to oscillate, going up a bit, going down a bit, but generally on an upward trend.

#!/usr/bin/perl
use threads;
use strict;
use warnings;

my $count=0;
for (;;) {

print $count++.' ';
my $thr = threads->new(
         sub { my $a = [ 1 .. 10000] }
                )->join;

 undef $thr;
}

[faxm0dem]

I tried using a similar OS (linux 2.6.9 x86_64) but perl 5.8.5, and the code seems to run OK.

[zentara]

Well, it seems you found another reason to be suspicious of threads, especially if you plan to use the script on multiple unknown platforms. On the other hand, it is a bit unusal to spawn 10,000 threads, and a 5 Meg/10000 threads isn't outrageous.

Many nodes have been posted concerning mem gains with threads, usually when complex objects (modules) are used in the threads, and the ref count dosn't clean up correctly. The solution there, is to reuse threads and objects, instead of spawning new ones. So....it seems the lesson is to reuse and recycle if you want to be safe with threads.