Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

does threads (still) memleak?

by faxm0dem (Novice)
on Nov 17, 2008 at 10:14 UTC ( #724026=perlquestion: print w/ replies, xml ) Need Help??
faxm0dem has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm having huge memleaks using threads; I came upon node_id=625622, which has no conclusion. Here's the thing: running the following code uses up more and more memory, but quite unpredictably depending on the contents of $sub1:
use threads; use strict; use warnings; for (;;) { my $thr = threads->new($sub1)->join; undef $thr; }
I'm monitoring rss usage on linux and perl 5.10 using:
i=0; while :; do A=$(ps -C threads3 -o ppid,pid,cmd,rss --no-headers); + [ -n "$A" ] && echo $i $A || i=0 ; sleep 1; i=$(($i+1)); done
Here's the output of this, with different values of $sub1:
# $sub1 = sub { my $a=[ 1 .. 10000] }; 1 19959 1137 /usr/bin/perl ./threads3 5276 2 19959 1137 /usr/bin/perl ./threads3 5140 3 19959 1137 /usr/bin/perl ./threads3 5700 [...] 642 19959 1137 /usr/bin/perl ./threads3 124244 643 19959 1137 /usr/bin/perl ./threads3 124736 644 19959 1137 /usr/bin/perl ./threads3 125380 # $sub1 = sub { my $a=[ 1 .. 100000] }; 1 19959 8329 /usr/bin/perl ./threads3 24176 2 19959 8329 /usr/bin/perl ./threads3 24176 3 19959 8329 /usr/bin/perl ./threads3 24176 [...] 642 19959 8329 /usr/bin/perl ./threads3 28368 643 19959 8329 /usr/bin/perl ./threads3 26036 644 19959 8329 /usr/bin/perl ./threads3 34460 # $sub1 = sub { }; 1 19959 13288 /usr/bin/perl ./threads3 3152 2 19959 13288 /usr/bin/perl ./threads3 3796 3 19959 13288 /usr/bin/perl ./threads3 4564 [...] 642 19959 13288 /usr/bin/perl ./threads3 400640 643 19959 13288 /usr/bin/perl ./threads3 401312 644 19959 13288 /usr/bin/perl ./threads3 401980
Any idea what's happening here? $sub1 = sub { my $a= 1 .. 100000 } seems to have minor memleak, while an empty sub does the most damage... I tried using 'forks' instead, which seems to solve the problem. PS: The code proposed by BrowserUk in 625622 also leaks

Comment on does threads (still) memleak?
Select or Download Code
Re: does threads (still) memleak?
by BrowserUk (Pope) on Nov 17, 2008 at 11:16 UTC

    This is either platform specific, or a bug introduced since 5.8.6 because your code above (with the addition of a counter to display how many threads have been created and joined), doesn't leak on my system using 5.8.6 and threads 1.71:

    use threads; use strict; use warnings; print $], ' ', $threads::VERSION, "\n"; my $sub1 = sub { my $a=[ 1 .. 10000] }; my $m = 0; for (;;) { printf "\r%d\t", $m++; my $thr = threads->new( $sub1 )->join; undef $thr; } __END__ c:\test>junk 5.008006 1.71 3077 Terminating on signal SIGINT(2)

    For proof, see this image. It shows the cpu, memory usage and IO activity for the process while the code above created and destroyed those 3000+ threads. The memory usage wobbles up and down a bit, but basically stays level at around 2.7 MB.

    I note also that in the same thread that you cited, that dave_the_m says that "the bug is fixed in bleed perl"?. (Unfortunately he didn't see fit to inform us of the cause, or the affected platforms:()

    So, you could try either reverting to 5.8.6 or possibly upgrading to recently announced the 5.8.9 release candidate.

    I tried using 'forks' instead, which seems to solve the problem.

    That's a pretty meaningless statement. It's like saying that taking the train cured your car's fuel consumption problem :)


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      I tried BrowserUk code on my Linux machine and 5.10 and saw no leak as well.
      I am using ActiveState build 1001 and CentOS release 4.5.
      I see the same as you with v5.8.7, ActivePerl Build 813, WinXP.

        Thanks. It's looking more and more like the suspicions I've had and voiced about 5.8.8 and threads are right. Something pretty fundemental broke in that build and has only recently been corrected.

        That why I stayed back at 5.8.6. (I forgot why I rejected 5.8.7--I seem to recall having some problem with it when it was first released--the installer or maybe PPM didn't work right).


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: does threads (still) memleak?
by zentara (Archbishop) on Nov 17, 2008 at 13:53 UTC
    UPDATE Also just tested this on the latest Ubuntu8.10, with Perl5.10 and latest threads.....same memory gain.

    I think you should look at OS memory reclamation with threads on linux. Depending on your kernel, OS, computer speed, available free memory, etc, memory reclamation will change. I would guess that with so much demand from the script, and with alot of free memory, the kernel may be playing it safe and not freeing all the time. You might try running it with a sleep 1 in your loop, and give the kernel time to free things up. I'm trying it now, bit it may take a while :-)

    I have perl 5.8.8, and the latest threads from cpan, running on linux. When the script below starts, its at 15M, and at count 10,000, it is up to 20M. On the way up it seems to oscillate, going up a bit, going down a bit, but generally on an upward trend.

    #!/usr/bin/perl use threads; use strict; use warnings; my $count=0; for (;;) { print $count++.' '; my $thr = threads->new( sub { my $a = [ 1 .. 10000] } )->join; undef $thr; }

    I'm not really a human, but I play one on earth Remember How Lucky You Are
      I tried using a similar OS (linux 2.6.9 x86_64) but perl 5.8.5, and the code seems to run OK.
        Well, it seems you found another reason to be suspicious of threads, especially if you plan to use the script on multiple unknown platforms. On the other hand, it is a bit unusal to spawn 10,000 threads, and a 5 Meg/10000 threads isn't outrageous.

        Many nodes have been posted concerning mem gains with threads, usually when complex objects (modules) are used in the threads, and the ref count dosn't clean up correctly. The solution there, is to reuse threads and objects, instead of spawning new ones. So....it seems the lesson is to reuse and recycle if you want to be safe with threads.


        I'm not really a human, but I play one on earth Remember How Lucky You Are

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://724026]
Approved by rovf
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (7)
As of 2014-07-28 11:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (196 votes), past polls