Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Strange memory leak using just threads

by MnkyBrain (Novice)
on Sep 19, 2010 at 20:09 UTC ( #860732=perlquestion: print w/ replies, xml ) Need Help??
MnkyBrain has asked for the wisdom of the Perl Monks concerning the following question:

I'm using perl v5.10.0 on a debian lenny system (perl package version 5.10.0-19lenny2) and I'm experiencing a very strange memory leak in a threaded program.

I started stripping down the code to try to find what was causing it (assuming the problem was coming from one of the modules I was using), and was pretty surprised when I got it down to just the threading code, and was still seeing the memory leak. Here's the code:

#! /usr/bin/perl use strict; use warnings; use threads; my @child; my @list = qw(a b c d); while (1) { foreach(@list){ push(@child,threads->create("test","$_")); } foreach my $ch (@child) { $ch->join(); } @child = (); } sub test { my ($item) = @_; }

As you can see, there is absolutely nothing going on other than thread creation/joining, yet this script leaks memory like a sieve on my machine.

Can anyone tell me what I've done wrong?

Comment on Strange memory leak using just threads
Download Code
Re: Strange memory leak using just threads
by Corion (Pope) on Sep 19, 2010 at 20:17 UTC

    This might be the same that happens in Perl bug #77854, as detailed on Google Groups. Maybe 5.10 has a bug there, or it depends on libpthreads. dave_the_m commented that he can't reproduce the problem on 5.10 or 5.12, so it might be interesting to see whether you can reproduce the problem on 5.12 (or bleadperl) too.

      My libpthreads seems to be 2.7 (or, at least, the libc6 package it is contained in is 2.7. Specifically, 2.7-18lenny4).

      I'll give 5.12 a shot and see what happens.

        I downloaded perl 5.12.2, and I'm still experiencing the memory leak. This is using the threads 1.75 that came with the distribution, not the 1.77 I installed for 5.10.

        Here are the options I used with configure:

        -des -Dprefix=$HOME/perl512 -Dusethreads -Duselargefiles -Dccflags=- +DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Doptimize=-O2 + -Duseshrplib -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ua +larm -Uusesfio -Uusenm -DDEBUGGING=-g

Re: Strange memory leak using just threads
by BrowserUk (Pope) on Sep 19, 2010 at 20:47 UTC

    Which version of threads?

    I don't see any leak using perl 5.010001 and threads: 1.76.

      My thread version seems to be 1.67. I upgraded to 1.77, and the problem still occurs.

      I did add print $threads::VERSION,"\n"; so I was sure it was using the correct module.

      I also tried it on another system with 5.8.8 installed and threads 1.07, and got the same result.

        This would appear to be a platform specific thing. Not particularly helpful, but it might allow you to look for a solution in the right place; eg. outside of Perl/threads.

        Perhaps there is a later version of the pthreads libraries for your system? Or known bugs within the version you have?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Strange memory leak using just threads
by zentara (Archbishop) on Sep 20, 2010 at 10:58 UTC

      I can run this for as long as I like:

      #! /usr/bin/perl use strict; use warnings; use threads; print "threads: ", $threads::VERSION; print "perl: ", $]; my @child; my @list = 1 .. 100; while (1) { my @child = map threads->create("test","$_"), @list; $_->join for @child; } sub test { my ($item) = @_; }

      And the memory usage is rock steady. And that has been true for a long time.

      So, what does that tell you?

      Your knowledge is out of date. Your platform--be it the pthreads or memory management or whatever--has an underlying problem that cannot be directly addressed by, nor attributed to, Perl. So stop doing that.

      BTW: Your "simple reusable threads demo", is way overcomplicated. And always was.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        And the memory usage is rock steady. And that has been true for a long time. So, what does that tell you? Your knowledge is out of date. Your platform--be it the pthreads or memory management or whatever--has an underlying problem that cannot be directly addressed by, nor attributed to, Perl. So stop doing that.

        So what does that tell me? First, his script gained memory and your's didn't. I observe the same behavior. So a script written 1 way in Perl may leak, and written another may not. Furthermore, these 2 sample thread scripts are the ultimate in simplicity. Leakage problems become more pronounced when you start adding objects to threaded programs.

        I have the latest Slackware, and before I believe my slackware c libs are faulty, I have to assume Perl's behavior is somehow causing it.

        Since this memory leak problem keeps coming up, all I can do is to mention that thread reuse does prevent the problem.

        But you are more of a thread expert than me, so if you insist that perl threads do not have memory problems, and it is the underlying c library's fault, I call that a cop out. If threads have no problems with ref counts and memleaks, why did you post threads: spawn early to avoid the crush.? Or is that information from you already outdated?


        I'm not really a human, but I play one on earth.
        Old Perl Programmer Haiku ................... flash japh
        Your platform--be it the pthreads or memory management or whatever--has an underlying problem that cannot be directly addressed by, nor attributed to, Perl.

        Strangely this problem doesn't affect C and Python threaded programs.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://860732]
Approved by moritz
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (9)
As of 2014-07-30 16:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (236 votes), past polls