Beefy Boxes and Bandwidth Generously Provided by pair Networks RobOMonk
There's more than one way to do things
 
PerlMonks  

memory leak with Thread::Queue ?

by Random_Walk (Parson)
on May 16, 2006 at 07:38 UTC ( #549677=perlquestion: print w/ replies, xml ) Need Help??
Random_Walk has asked for the wisdom of the Perl Monks concerning the following question:

Morning all

I am having problems with my multithreaded tcp listener with IO::Socket. It is leaking memory even after applying the good advice from BrowserUk. I suspect it may be a problem with the Thread::Queue. Here is a simple script that shows the problem (unixy 'cos of ps command, strip that out and use taskmangler to see it on windows).

#!/usr/bin/perl use strict; use warnings; use threads; use Thread::Queue; my $queue = Thread::Queue -> new; threads -> create ("popper", $queue) -> detach for (1..10); my $message="this is a line of test around the right size"; while (1) { $queue -> enqueue ($message) for (1..100); sleep 1 while $queue -> pending; print $/,`ps -eo vsize,args | grep $0 | grep -v grep`; } sub popper { print "started a popper\n"; my $queue = shift; while ($queue -> dequeue) { print "."; } }
Here is a snatch of output ...
6300 /usr/bin/perl ./threadtest.pl ...................................................................... +.............................. 6316 /usr/bin/perl ./threadtest.pl ...................................................................... +.............................. 6340 /usr/bin/perl ./threadtest.pl ...................................................................... +.............................. 6364 /usr/bin/perl ./threadtest.pl ...................................................................... +.............................. 6408 /usr/bin/perl ./threadtest.pl ...................................................................... +..............................

Is this because a variable on the queue is being made shared between all threads and never going out of scope ? If so is this fixable or an imutable feature of thread queues ?

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Comment on memory leak with Thread::Queue ?
Select or Download Code
Re: memory leak with Thread::Queue ?
by dave_the_m (Parson) on May 16, 2006 at 09:05 UTC
    What version of perl are you using? I believe that leak was fixed in 5.8.1

    Dave.

      Hi Dave_the_m

      I think you have hit the nail on the head

      >perl -v This is perl, v5.8.0 built for aix-thread-multi Copyright 1987-2002, Larry Wall Perl may be copied only under the terms of either the Artistic License + or the GNU General Public License, which may be found in the Perl 5 source ki +t. Complete documentation for Perl, including FAQ lists, should be found +on this system using `man perl' or `perldoc perl'. If you have access to + the Internet, point your browser at http://www.perl.com/, the Perl Home Pa +ge.
      I will test it on some later flavours.

      Update

      Sure enough a test on 5.8.7 works fine as well. Version to blame, sorry for the inconvenience

      Many thanks,
      R.

      Pereant, qui ante nos nostra dixerunt!
Re: memory leak with Thread::Queue ?
by BrowserUk (Pope) on May 16, 2006 at 10:04 UTC

    Your code (slightly modified for win32), doesn't appear to leak under AS811/5.8.6 or AS817/5.8.8.

    #!/usr/bin/perl use strict; use warnings; use threads; use Thread::Queue; my $queue = Thread::Queue -> new; threads -> create ("popper", $queue) -> detach for (1..10); my $message="this is a line of test around the right size"; while (1) { $queue -> enqueue ($message) for (1..100); sleep 1 while $queue -> pending; system qq[tasklist /nh /fi "PID eq $$"]; } sub popper { print "started a popper\n"; my $queue = shift; while ($queue -> dequeue) { print "."; } } __END__ C:\test>junk5 started a popper started a popper started a popper started a popper started a popper started a popper started a popper started a popper started a popper started a popper perl.exe 3128 0 6,936 +K perl.exe 3128 0 6,944 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K perl.exe 3128 0 6,948 +K

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      This problem is present on RHEL5 running 5.10.0, and RHEL5 running 5.10.1. It is NOT present on RHEL4 running 5.10.0. I don't understand why.

        I'd suggest you re-post this under dave_the_m's post above (or take it directly to p5p), as he understands the problem and is best place to suggest how to fix it.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://549677]
Approved by davido
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (6)
As of 2014-04-18 08:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (464 votes), past polls