http://www.perlmonks.org?node_id=625622

misc has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I got in some trouble with the memory consumption of a threaded tcp server.
I created a new thread for every connect, and destroyed the thread after the connection has been closed.
However, I got an "Out of memory" after creating and destroying a few 100 threads, so I tried to figure out what's wrong.

I wrote a testscript, which constantly creates and exits threads, it's below.
This testscript will also run out of memory, but the memory consumption is quite unpredictable and seems to raise in steps.
Changing the sleeping time below the comment # SLEEP does have some effect, as longer the sleep time is here as slower the memory consumption raises.

Is there some way to avoid the memory leak ?

Thanks, Michael
#!/usr/bin/perl -w use threads; use threads::shared; share $threadscount; $threadscount = 0; share $maxthreads; $maxthreads = 0; share $threadscreated; $threadscreated = 0; # Dump out virtual memory and resident memory consumption sub printmemorysizes{ open F, "</proc/self/status"; my @s = <F>; close F; my @vmsize = grep /^VmSize.*/, @s; chomp $vmsize[0]; print $vmsize[0], " "; my @rss = grep /^VmRSS/, @s; print $rss[0]; } # The treads' sub sub thread{ my $count; { lock $threadscount; $threadscount ++; # count how many threads there are $count = $threadscount; lock $maxthreads; if ( $count>$maxthreads ){ $maxthreads = $count; # Store the maximum of concurrent threads } lock $threadscreated; $threadscreated ++; } #print "threadscreated: $threadscreated "; #&printmemorysizes(); select undef,undef,undef,1; lock $threadscount; $threadscount --; } while ( 1 ){ for ( 1..300 ){ my $t = threads->create("thread"); $t->detach(); } my $tc; do { select undef,undef,undef,0.1; # SLEEP # as shorter the sleep the memory cunsumption raises faster it seems lock $threadscount; $tc = $threadscount; } while ( $tc ); #sleep 2; # dump out: #maximum concurrent threads | How many threads have been created | # virtual memory size | resident memory size print "maxthreads: $maxthreads threadscreated: $threadscreated "; printmemorysizes(); $maxthreads = 0; }
------------- Some example output (nothing changed between the runs)
micha@laptop ~/prog/perl/test $ ./threads_leak.pl maxthreads: 300 threadscreated: 300 VmSize: 177468 kB VmRSS: + 125216 kB maxthreads: 300 threadscreated: 600 VmSize: 291888 kB VmRSS: + 248072 kB maxthreads: 300 threadscreated: 900 VmSize: 326356 kB VmRSS: + 248520 kB maxthreads: 300 threadscreated: 1200 VmSize: 426424 kB VmRSS: + 249172 kB maxthreads: 300 threadscreated: 1500 VmSize: 287920 kB VmRSS: + 252020 kB maxthreads: 300 threadscreated: 1800 VmSize: 287920 kB VmRSS: + 252464 kB maxthreads: 300 threadscreated: 2100 VmSize: 313928 kB VmRSS: + 252736 kB maxthreads: 300 threadscreated: 2400 VmSize: 1014284 kB VmRSS: + 253800 kB maxthreads: 300 threadscreated: 2700 VmSize: 525608 kB VmRSS: + 372676 kB micha@laptop ~/prog/perl/test $ ./threads_leak.pl maxthreads: 300 threadscreated: 300 VmSize: 161832 kB VmRSS: + 125272 kB maxthreads: 300 threadscreated: 600 VmSize: 1873224 kB VmRSS: + 128360 kB maxthreads: 300 threadscreated: 900 VmSize: 288048 kB VmRSS: + 248756 kB maxthreads: 300 threadscreated: 1200 VmSize: 1367664 kB VmRSS: + 250560 kB maxthreads: 300 threadscreated: 1500 VmSize: 370560 kB VmRSS: + 249440 kB maxthreads: 300 threadscreated: 1800 VmSize: 724704 kB VmRSS: + 250108 kB maxthreads: 300 threadscreated: 2100 VmSize: 410060 kB VmRSS: + 372136 kB

Replies are listed 'Best First'.
Re: memory leaks with threads
by dave_the_m (Monsignor) on Jul 09, 2007 at 15:22 UTC
    This leak appears to be fixed in bleedperl (5.9.5).

    Dave.

      I just compiled and installed perl 5.9.5.
      The memory consumption raises a bit slower now, but still raises.

      In the hope it's useful here's the output of perl -V.
      Summary of my perl5 (revision 5 version 9 subversion 5) configuration: Platform: osname=linux, osvers=2.6.18-32-splash, archname=i686-linux-thread- +multi uname='linux laptop 2.6.18-32-splash #3 smp preempt mon feb 5 14:1 +2:53 local time zone must be set--s i686 amd turion(tm) 64 x2 mobile +technology tl-60 authenticamd gnulinux ' config_args='-Dprefix=/home/micha/localperl -Dusethreads -des -Dus +edevel' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=und +ef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing + -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=6 +4', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I +/usr/local/include' ccversion='', gccversion='4.1.1 (Gentoo 4.1.1-r1)', gccosandvers=' +' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -l +c perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.4.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.4' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Built under linux Compiled at Jul 9 2007 18:20:04 @INC: /home/micha/localperl/lib/5.9.5/i686-linux-thread-multi /home/micha/localperl/lib/5.9.5 /home/micha/localperl/lib/site_perl/5.9.5/i686-linux-thread-multi /home/micha/localperl/lib/site_perl/5.9.5
Re: memory leaks with threads
by BrowserUk (Patriarch) on Jul 09, 2007 at 15:58 UTC
      perl 5.8.8,
      threads 1.07
      threads::shared 0.94

      I just upgraded threads and threads::shared to 1.63 and 1.12,
      but this didn't change anything.

      At the moment I'm trying to compile perl 5.9.5..

        The thing is, I'm not at all convinced that your posted code is evidence of a memory leak.

        On the surface, it looks like it is because it looks like you are recording the maximum number of concurrent threads, but the way you are doing it is flawed.

        If your main thread gets a timeslice between this line:

        $count = $threadscount;

        and this line:

        if ( $count>$maxthreads ){

        Then it could easily create 300 more threads, that would increment $threadscount, but they would not be reflected in $count, so you would not be recording the true maximum concurrent threads.

        Basically, there is nothing in your code to prevent you from creating more than 300 threads, so the fact that the maximum thread count never gets higher than 300 is highly suspicious.

        In essence, I don't believe you are demonstrating a memory leak. I think the amount of memory used is simply a multiplication factor resulting from the fact that there is no constraint on the concurrent number of threads, and at various times, this number is rising well above the 300 number that your code is reporting.

        Would you please run the following code, which explicitly constrains the maximum number of threads, whilst creating and destroying threads at a high rate and monitor the memory usage using top and post your findings?

        use threads; use threads::shared; our $S ||= 1; my $t:shared = 0; $|++; while( 1 ){ printf"\r$m\t%d\t", do{ lock $t; $t }; if( do{ lock $t; $t } < 10 ) { $m++; threads->create( sub{ { lock $t; $t++ } select undef, undef, undef, $S; { lock $t; $t-- } } )->detach; } else{ select undef, undef, undef, 0.01; } }

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.