Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Perl Threads and Memory Leaking

by aphexcoil (Novice)
on Oct 06, 2009 at 05:10 UTC ( #799383=perlquestion: print w/replies, xml ) Need Help??
aphexcoil has asked for the wisdom of the Perl Monks concerning the following question:

Perhaps someone far wiser than I can explain why this:
#!/usr/bin/perl use Thread; use Time::HiRes qw(usleep); while(usleep(100)) { my $thr1 = new Thread \&doNothing1; $thr1->join; } sub doNothing {}
... eats memory? I ran this and watched TOP and noticed that the memory perl was using kept increasing. I thought join cleaned up after a thread?

Replies are listed 'Best First'.
Re: Perl Threads and Memory Leaking
by BrowserUk (Pope) on Oct 06, 2009 at 06:37 UTC

    On win, I see a slow creep up of memory usage for the first few thousand iterations, but it then stabilises and stays rock steady for tens of thousands more:

    #!/usr/bin/perl use Thread; use Time::HiRes qw(usleep); my $count = 0; while( 1 ) { usleep(100); my $thr1 = new Thread \&doNothing; $thr1->join; unless( ++$count % 1000 ) { printf "%3d: %s", $count, ( qx[tasklist /fo list /fi "pid eq $$"] )[ 5 ]; } } sub doNothing {} __END__ C:\test>junk 1000: Mem Usage: 11,576 K 2000: Mem Usage: 11,752 K 3000: Mem Usage: 11,880 K 4000: Mem Usage: 11,944 K 5000: Mem Usage: 12,544 K 6000: Mem Usage: 12,544 K 7000: Mem Usage: 12,544 K 8000: Mem Usage: 12,544 K 9000: Mem Usage: 12,544 K 10000: Mem Usage: 12,544 K 11000: Mem Usage: 12,544 K 12000: Mem Usage: 12,544 K 13000: Mem Usage: 12,544 K 14000: Mem Usage: 12,544 K 15000: Mem Usage: 12,544 K 16000: Mem Usage: 12,544 K 17000: Mem Usage: 12,544 K 18000: Mem Usage: 12,544 K 19000: Mem Usage: 12,544 K 20000: Mem Usage: 12,544 K 21000: Mem Usage: 12,544 K 22000: Mem Usage: 12,544 K 23000: Mem Usage: 12,544 K 24000: Mem Usage: 12,544 K 25000: Mem Usage: 12,544 K 26000: Mem Usage: 12,544 K 27000: Mem Usage: 12,544 K 28000: Mem Usage: 12,544 K 29000: Mem Usage: 12,544 K 30000: Mem Usage: 12,544 K 31000: Mem Usage: 12,544 K 32000: Mem Usage: 12,544 K 33000: Mem Usage: 12,544 K Terminating on signal SIGINT(2)

    If there's a leak, it appears to be platform specific.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Perl Threads and Memory Leaking
by ELISHEVA (Prior) on Oct 06, 2009 at 08:00 UTC

    Memory consumption seems to grow for Perl 5.8.8 running on Debian Etch as well.

    use strict; use warnings; use Thread; use Time::HiRes qw(usleep); my $count = 0; print " VIRT RES SHR S %CPU %MEM TIME+\n"; while( 1 ) { usleep(100); my $thr1 = new Thread \&doNothing; $thr1->join; unless( ++$count % 1000 ) { printf "%-10s%4s %3s %3s %s %4s %4s %s\n" , "$count: " , map { s/^\s+//; (split(/\s+/,$_,-1))[4..10] } grep {/^\s*$$/} qx{top -n 1 -p $$ -b}; } } sub doNothing {} __END__ VIRT RES SHR S %CPU %MEM TIME+ 1000: 13536 3784 1488 S 0.0 1.5 0:01.03 2000: 14256 4504 1492 S 0.0 1.8 0:11.59 3000: 14980 5220 1492 S 0.0 2.0 0:12.27 4000: 15700 5936 1492 S 0.0 2.3 0:20.39 5000: 16420 6652 1492 S 0.0 2.6 0:30.56 6000: 17144 7368 1492 S 0.0 2.9 0:40.29 7000: 17864 8084 1492 S 0.0 3.2 0:41.38 8000: 18504 8800 1492 S 0.0 3.4 0:42.78 9000: 19224 9516 1492 S 0.0 3.7 0:46.31 10000: 19944 9m 1492 S 0.0 4.0 0:50.17 11000: 20664 10m 1492 S 0.0 4.3 0:52.94 12000: 21388 11m 1492 S 0.0 4.5 0:53.33 13000: 22108 12m 1492 S 0.0 4.8 0:54.48 14000: 22828 12m 1492 S 0.0 5.1 1:00.93 15000: 23552 13m 1492 S 0.0 5.4 1:13.55 16000: 24272 14m 1492 S 0.0 5.7 1:23.45 17000: 24992 14m 1492 S 0.0 5.9 1:30.25 18000: 25712 15m 1492 S 0.0 6.2 1:30.35 19000: 26432 16m 1492 S 0.0 6.5 1:38.96 20000: 27152 16m 1492 S 0.0 6.8 1:49.57 21000: 27872 17m 1492 S 0.0 7.1 1:59.23 22000: 28592 18m 1492 S 2.0 7.3 1:59.82 23000: 29312 19m 1492 S 0.0 7.6 2:02.34 24000: 29956 19m 1492 S 0.0 7.9 2:09.54 25000: 30676 20m 1492 S 0.0 8.2 2:18.27 26000: 31396 21m 1492 S 0.0 8.5 2:21.90 27000: 32116 21m 1492 S 0.0 8.7 2:30.96 28000: 32836 22m 1492 S 0.0 9.0 2:33.76 29000: 33556 23m 1492 S 0.0 9.3 2:34.37 30000: 34276 23m 1492 S 0.0 9.6 2:37.70 31000: 34996 24m 1492 S 2.0 9.9 2:46.51 32000: 35716 25m 1492 S 0.0 10.1 2:56.34 33000: 36436 26m 1492 S 0.0 10.4 3:06.34 34000: 37156 26m 1492 S 0.0 10.7 3:14.56 35000: 37876 27m 1492 S 0.0 11.0 3:22.40 ... 163000: 126m 116m 1492 S 2.0 46.7 16:06.57 164000: 127m 117m 1492 S 0.0 47.0 16:08.22

    Best, beth

    Update Modified code so that number of fields in top output is consistent (leading spaces were making some runs start w/ field 4 and some with field 5).

Re: Perl Threads and Memory Leaking
by jakobi (Pilgrim) on Oct 06, 2009 at 08:51 UTC

    This one's even uglier:

    64bit jaunty 5.10.0 (5.10.0-19ubuntu1.1; most recent 'official' pkg) # -1 removed from the split in Beth' code #bash1 % perl ./x VIRT RES SHR S %CPU %MEM TIME+ 1000: 42796 8596 2048 S 0 0.2 0:02.80 2000: 47276 12m 2052 S 0 0.3 0:05.73 3000: 51708 17m 2052 S 0 0.4 0:08.69 4000: 56240 21m 2052 S 0 0.6 0:11.57 5000: 60704 25m 2052 S 0 0.7 0:14.65 6000: 65128 30m 2052 S 2 0.8 0:17.59 7000: 69636 34m 2052 S 0 0.9 0:20.69 8000: 74104 38m 2052 S 0 1.0 0:23.75 9000: 78556 43m 2052 S 0 1.1 0:26.89 10000: 83064 47m 2052 S 0 1.2 0:29.87 11000: 87564 52m 2052 S 0 1.3 0:32.92 12000: 92012 56m 2052 S 0 1.5 0:36.01 13000: 96540 60m 2052 S 0 1.6 0:39.11 14000: 98.7m 65m 2052 S 0 1.7 0:42.23 15000: 103m 69m 2052 S 0 1.8 0:45.49 16000: 107m 74m 2052 S 0 1.9 0:48.45 17000: 111m 78m 2052 S 0 2.0 0:51.40 18000: 116m 82m 2052 S 0 2.1 0:54.27 19000: 120m 87m 2052 S 0 2.2 0:57.23 20000: 124m 91m 2052 S 0 2.4 1:00.13 21000: 129m 95m 2052 S 0 2.5 1:03.08 22000: 133m 100m 2052 S 0 2.6 1:06.08 23000: 138m 104m 2052 S 0 2.7 1:09.07 24000: 142m 109m 2052 S 0 2.8 1:12.13 25000: 146m 113m 2052 S 0 2.9 1:15.17 26000: 151m 117m 2052 S 0 3.0 1:18.40 27000: 155m 122m 2052 S 0 3.1 1:21.46 28000: 159m 126m 2052 S 0 3.2 1:24.41 29000: 164m 130m 2052 S 0 3.4 1:27.35 30000: 168m 135m 2052 S 0 3.5 1:30.27 31000: 173m 139m 2052 S 0 3.6 1:33.16 32000: 177m 144m 2052 S 0 3.7 1:36.05 33000: 181m 148m 2052 S 0 3.8 1:38.94
      Right. This is exactly what I saw originally on a Redhat server at work. I then took the code and ran it on two home servers (Linix Mint and Ubuntu). At first I thought it was some data structures that weren't being destroyed. This affected both v5.8.8, v5.10.0 and v5.10.1 I'll have to work around this problem and design the logic so that the threads never die. My program only uses three threads max (well four if you count the parent) -- but at the rate of creating and joining those threads every few seconds, the memory leak would eventually consume all available memory. I am curious if this is a bug -- I'd really like to know where that memory is going.
Re: Perl Threads and Memory Leaking
by ikegami (Pope) on Oct 06, 2009 at 05:32 UTC
    After fixing the extraneous "1", I get lots of fluctuation in memory, but no increase. What version of Perl are you using?
      Perl v5.10.0 on Ubuntu Linux 32bit Extraneous 1?

        I don't get any increase with ActivePerl 5.10.0 or .1 on Windows.

        You should start trying with 5.10.1. Lots of stuff was fixed in 5.10.1

      Oh yeah ... sorry about that. That "1" should not be in the first reference to the subroutine.
Re: Perl Threads and Memory Leaking
by jakobi (Pilgrim) on Oct 06, 2009 at 14:43 UTC

    Data point alpha netbsd 2.1 perl 5.10.0

    (just remembered that I've access to a box differing a bit from the setups tested above. First col is actually SIZE, 2nd is RES; running niced and with lowered count intervals to be more courteous to the hoster)

    VIRT RES SHR S %CPU %MEM TIME+ 333: 5880K 7024K piperd 0:07 0.00% 26.37% perl 666: 7912K 9056K piperd 0:14 95.26% 43.02% perl 999: 9936K 11M piperd 0:21 120.78% 54.54% perl 1332: 12M 13M piperd 0:28 98.42% 60.40% perl 1665: 14M 15M piperd 0:36 99.62% 66.50% perl 1998: 16M 17M piperd 0:43 89.06% 67.14% perl 2331: 18M 19M piperd 0:50 82.70% 66.85% perl 2664: 20M 21M piperd 0:58 73.58% 63.13% perl 2997: 22M 23M piperd 1:05 76.28% 68.26% perl 3330: 24M 25M piperd 1:13 79.44% 71.09% perl 3663: 26M 27M piperd 1:20 76.92% 72.71% perl 3996: 27M 29M piperd 1:28 73.27% 68.36% perl 4329: 29M 31M piperd 1:35 62.32% 60.45% perl ... 8658: 55M 56M piperd 3:19 65.26% 65.23% perl 8991: 57M 58M piperd 3:28 66.03% 66.02% perl 9324: 59M 60M piperd 3:36 67.69% 67.68% perl ^c as I don't want to hog that old server for too long.

    It affects Linux and at least one BSD (Alpha, 64bit AFAIR; not just Intel...), both 32 and 64bit for Linux; with Perls for these hosts ranging from 5.8.8 to at least 5.10.1..

    Not nice.

      I can't find a linux box with a recent perl version where this doesn't occur. Perhaps submitting a bug report is in order. Perhaps we can get some clarification.
      I submitted a bug report. I'll let you know what I get for a reply. Thanks.

        Getting exactly the same with perl 5.10.1 on debian-testing

        1 7216 15735 /usr/bin/perl ./perl_thread 5948 2 7216 15735 /usr/bin/perl ./perl_thread 6500 3 7216 15735 /usr/bin/perl ./perl_thread 6072 4 7216 15735 /usr/bin/perl ./perl_thread 6668 5 7216 15735 /usr/bin/perl ./perl_thread 6868 6 7216 15735 /usr/bin/perl ./perl_thread 7716 7 7216 15735 /usr/bin/perl ./perl_thread 8204 8 7216 15735 /usr/bin/perl ./perl_thread 7796 9 7216 15735 /usr/bin/perl ./perl_thread 7888 10 7216 15735 /usr/bin/perl ./perl_thread 9220 11 7216 15735 /usr/bin/perl ./perl_thread 9556 12 7216 15735 /usr/bin/perl ./perl_thread 9664 13 7216 15735 /usr/bin/perl ./perl_thread 9680 14 7216 15735 /usr/bin/perl ./perl_thread 10548 15 7216 15735 /usr/bin/perl ./perl_thread 10320 16 7216 15735 /usr/bin/perl ./perl_thread 10960 17 7216 15735 /usr/bin/perl ./perl_thread 11568 18 7216 15735 /usr/bin/perl ./perl_thread 11772 19 7216 15735 /usr/bin/perl ./perl_thread 12232 20 7216 15735 /usr/bin/perl ./perl_thread 12584

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://799383]
Approved by ELISHEVA
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (7)
As of 2018-09-22 22:57 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Eventually, "covfefe" will come to mean:













    Results (190 votes). Check out past polls.

    Notices?
    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!