misc has asked for the wisdom of the Perl Monks concerning the following question:
Hi,
I got in some trouble with the memory consumption of a threaded tcp server.
I created a new thread for every connect, and destroyed the thread after the connection has been closed.
However, I got an "Out of memory" after creating and destroying a few 100 threads, so I tried to figure out what's wrong.
I wrote a testscript, which constantly creates and exits threads, it's below.
This testscript will also run out of memory, but the memory consumption is quite unpredictable and seems to raise in steps.
Changing the sleeping time below the comment # SLEEP does have some effect, as longer the sleep time is here as slower the memory consumption raises.
Is there some way to avoid the memory leak ?
Thanks, Michael
I got in some trouble with the memory consumption of a threaded tcp server.
I created a new thread for every connect, and destroyed the thread after the connection has been closed.
However, I got an "Out of memory" after creating and destroying a few 100 threads, so I tried to figure out what's wrong.
I wrote a testscript, which constantly creates and exits threads, it's below.
This testscript will also run out of memory, but the memory consumption is quite unpredictable and seems to raise in steps.
Changing the sleeping time below the comment # SLEEP does have some effect, as longer the sleep time is here as slower the memory consumption raises.
Is there some way to avoid the memory leak ?
Thanks, Michael
------------- Some example output (nothing changed between the runs)#!/usr/bin/perl -w use threads; use threads::shared; share $threadscount; $threadscount = 0; share $maxthreads; $maxthreads = 0; share $threadscreated; $threadscreated = 0; # Dump out virtual memory and resident memory consumption sub printmemorysizes{ open F, "</proc/self/status"; my @s = <F>; close F; my @vmsize = grep /^VmSize.*/, @s; chomp $vmsize[0]; print $vmsize[0], " "; my @rss = grep /^VmRSS/, @s; print $rss[0]; } # The treads' sub sub thread{ my $count; { lock $threadscount; $threadscount ++; # count how many threads there are $count = $threadscount; lock $maxthreads; if ( $count>$maxthreads ){ $maxthreads = $count; # Store the maximum of concurrent threads } lock $threadscreated; $threadscreated ++; } #print "threadscreated: $threadscreated "; #&printmemorysizes(); select undef,undef,undef,1; lock $threadscount; $threadscount --; } while ( 1 ){ for ( 1..300 ){ my $t = threads->create("thread"); $t->detach(); } my $tc; do { select undef,undef,undef,0.1; # SLEEP # as shorter the sleep the memory cunsumption raises faster it seems lock $threadscount; $tc = $threadscount; } while ( $tc ); #sleep 2; # dump out: #maximum concurrent threads | How many threads have been created | # virtual memory size | resident memory size print "maxthreads: $maxthreads threadscreated: $threadscreated "; printmemorysizes(); $maxthreads = 0; }
micha@laptop ~/prog/perl/test $ ./threads_leak.pl maxthreads: 300 threadscreated: 300 VmSize: 177468 kB VmRSS: + 125216 kB maxthreads: 300 threadscreated: 600 VmSize: 291888 kB VmRSS: + 248072 kB maxthreads: 300 threadscreated: 900 VmSize: 326356 kB VmRSS: + 248520 kB maxthreads: 300 threadscreated: 1200 VmSize: 426424 kB VmRSS: + 249172 kB maxthreads: 300 threadscreated: 1500 VmSize: 287920 kB VmRSS: + 252020 kB maxthreads: 300 threadscreated: 1800 VmSize: 287920 kB VmRSS: + 252464 kB maxthreads: 300 threadscreated: 2100 VmSize: 313928 kB VmRSS: + 252736 kB maxthreads: 300 threadscreated: 2400 VmSize: 1014284 kB VmRSS: + 253800 kB maxthreads: 300 threadscreated: 2700 VmSize: 525608 kB VmRSS: + 372676 kB micha@laptop ~/prog/perl/test $ ./threads_leak.pl maxthreads: 300 threadscreated: 300 VmSize: 161832 kB VmRSS: + 125272 kB maxthreads: 300 threadscreated: 600 VmSize: 1873224 kB VmRSS: + 128360 kB maxthreads: 300 threadscreated: 900 VmSize: 288048 kB VmRSS: + 248756 kB maxthreads: 300 threadscreated: 1200 VmSize: 1367664 kB VmRSS: + 250560 kB maxthreads: 300 threadscreated: 1500 VmSize: 370560 kB VmRSS: + 249440 kB maxthreads: 300 threadscreated: 1800 VmSize: 724704 kB VmRSS: + 250108 kB maxthreads: 300 threadscreated: 2100 VmSize: 410060 kB VmRSS: + 372136 kB
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: memory leaks with threads
by dave_the_m (Monsignor) on Jul 09, 2007 at 15:22 UTC | |
by misc (Friar) on Jul 09, 2007 at 16:48 UTC | |
by BrowserUk (Patriarch) on Jul 09, 2007 at 15:38 UTC | |
Re: memory leaks with threads
by BrowserUk (Patriarch) on Jul 09, 2007 at 15:58 UTC | |
by misc (Friar) on Jul 09, 2007 at 16:15 UTC | |
by BrowserUk (Patriarch) on Jul 09, 2007 at 16:52 UTC | |
by misc (Friar) on Jul 09, 2007 at 17:39 UTC | |
by BrowserUk (Patriarch) on Jul 09, 2007 at 18:24 UTC | |
| |
by royer_k (Initiate) on Sep 24, 2007 at 10:55 UTC | |
by BrowserUk (Patriarch) on Sep 24, 2007 at 12:42 UTC |
Back to
Seekers of Perl Wisdom