Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

memory leak with perl ithreads

by franzs (Initiate)
on Apr 03, 2011 at 09:14 UTC ( #897173=perlquestion: print w/ replies, xml ) Need Help??
franzs has asked for the wisdom of the Perl Monks concerning the following question:

Dear Perlmonks, I wrote a (larger) script using perl 5.10.1 and ithreads. This script creates new threads at a certain rate to achieve a asynchronous execution. Although the threads are joined and only a specific number of threads is allowed to run at a time the memory usage of the perl interpreter process increases over time. This increase is more or less linear:
time/s virtual residential 00:00:00 16504 8428 00:01:00 17528 9468 00:02:00 17656 10292 00:03:00 18680 10740 00:04:00 18680 11536 00:05:00 19704 12228 00:06:00 20728 12916 00:07:00 21752 13812 00:08:00 21752 14288 00:09:00 23800 14964 00:10:00 23800 16108 00:11:00 23800 16380 00:12:00 24824 17168 00:13:00 25848 17912 00:14:00 26872 18760 00:15:00 26872 19228 00:16:00 27896 20164 00:17:00 28920 20860 00:18:00 29944 21688 00:19:00 30968 22592 00:20:00 30968 23328 00:21:00 31992 24060 00:22:00 31992 24696 00:23:00 33016 25328 00:24:00 34040 26140 00:25:00 35064 26812 00:26:00 35064 27780 00:27:00 36088 28340 00:28:00 37112 29272 00:29:00 38136 30084 00:30:00 38136 30856 00:31:00 39160 31368 00:32:00 40184 32268 00:33:00 41208 33032
Any idea how to prevent the script from using more and more memory? The script below is a stripped down version of the original script:
#!/usr/bin/perl use strict; use warnings; use Config; use threads; use Time::HiRes qw( usleep ); $Config{useithreads} or die('Recompile Perl with threads to run this p +rogram.'); foreach my $i (1 .. 100000) { while ((my $nthreads = scalar threads->list()) >= 10) { print $nthreads . " running. waiting 1 second.\n"; sleep(1); foreach my $thread (threads->list(threads::joinable)) +{ $thread->join(); } } threads->create(\&sub1); usleep(333333); foreach my $thread (threads->list(threads::joinable)) { $thread->join(); } } foreach my $thread (threads->list()) { $thread->join(); } sub sub1 { my $sleep_value = int(rand(2) * 100) / 100; print "thread id " . threads->tid() . ": Start sleeping " . $s +leep_value . " seconds\n"; usleep($sleep_value * 1000000); print "thread id " . threads->tid() . ": Done sleeping " . $s +leep_value . " seconds\n"; }
Details of perl interpreter:
Summary of my perl5 (revision 5 version 10 subversion 1) configuration +: Platform: osname=freebsd, osvers=8.2-release, archname=i386-freebsd-thread-m +ulti-64int uname='freebsd hal.hsh 8.2-release freebsd 8.2-release #0: wed feb + 23 11:54:06 cet 2011 dirk@hal.hsh:exportobjusrsrcsyshal i386 ' config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/per +l5/5.10.1/mach -Dprivlib=/usr/local/lib/perl5/5.10.1 -Dman3dir=/usr/l +ocal/lib/perl5/5.10.1/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Ds +itearch=/usr/local/lib/perl5/site_perl/5.10.1/mach -Dsitelib=/usr/loc +al/lib/perl5/site_perl/5.10.1 -Dscriptdir=/usr/local/bin -Dsiteman3di +r=/usr/local/lib/perl5/5.10.1/man/man3 -Dsiteman1dir=/usr/local/man/m +an1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Di +nc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5. +10.1/BSDPAN" -Doptimize=-O2 -pipe -fno-strict-aliasing -Ud_dosuid -Ui +_gdbm -Dusethreads=y -Dusemymalloc=n -Duse64bitint -Dusemultiplicity= +y' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=und +ef use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: [...]

Comment on memory leak with perl ithreads
Select or Download Code
Re: memory leak with perl ithreads
by Anonymous Monk on Apr 03, 2011 at 10:14 UTC
    No leak on win32 5.12.2, "Mem Usage" stays under 7k, "VM Size" under 5k, up to thread id 1863
Re: memory leak with perl ithreads
by dave_the_m (Parson) on Apr 03, 2011 at 11:12 UTC
    I don't see a leak on linux with 5.10.1. Perhaps it's a leak in the BSD threads library?

    Dave.

      following up to myself: it does indeed leak on 5.10.1 under linux, but only on non-debugging builds (I should have remembered that). Fixed in 5.13.5

      Dave.

        Is 5.14 still on schedule for the 8th April release date?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: memory leak with perl ithreads
by zwon (Monsignor) on Apr 03, 2011 at 11:17 UTC

    If you search for "thread leak" on perlmonks, you find the answer, the problem was discussed many times here. Shortly, on *nix perl threads leaking memory every time you creating a new thread. The bug was fixed somewhere around 5.13.9 if I remember correctly.

    You should not create many threads, thread creation is rather expensive operation (on *nix it is more expensive than fork, I'm talking about perl threads of course). Solution is to use pool of threads.

        Yes, win is not affected
Re: memory leak with perl ithreads
by Anonymous Monk on Nov 10, 2012 at 02:29 UTC
    I've also had this problem on CentOS. It seems that usleep() is causing the memory leak. For some reason usleep() seems to kill the httpd script, and httpd processes (as indicated by the "top" shell command) eventually overload the server. After removing usleep() from my script, no more memory leaks.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://897173]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (12)
As of 2014-09-20 13:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (159 votes), past polls