Re^3: ithreads memory leak

[BrowserUk]

I don't believe you are seeing a leak. I have to express it this way because I don't have *nix to test on; but running your code on my system, I see memory growth, but not a leak.

To confirm my findings, tweak your code so:

#!/home/alerting/perl5/perlbrew/perls/perl-5.20.2/bin/perl

use warnings;
use IO::Handle;
use threads stack_size => 64*1024;
use strict;
use DBI;
use Email::MIME;
use Email::Sender::Simple qw(sendmail);

while (<STDIN>)
{
     threads->create(\&thread)->detach;
     sleep 5;
}

sub thread
{
        print "Thread started!\n";
        sleep 4;
}

Essentially what that does is limit your program to using only child thread at a time. (Not useful in the real world, but good for testing.).

Run that and hold the enter key down while monitoring the memory usage and you should see an initial jump from circa 7MB to around 13MB, and then the memory usage should stay steady no matter how long you hold the key down for. Come back and let me know if that is true; and then I'll try and explain what you are seeing.

[DNAb]

With the above code the usage stays steady at 23MB, compared to the hundreds of MBs. Certainly curious to hear how to deal with this, I will indeed need multiple child threads running. I thought that PERL should be doing garbage collection once the threads terminate.

[BrowserUk]

I thought that PERL should be doing garbage collection once the threads terminate.

It does. But -- unlike fork processes -- that memory is returned to the process free memory pool, not to the operating system.

That is, the memory used by a ended thread is freed, and can be reused for the next thread -- that is why when you only use one thread at a time, the memory used by the process stays steady. The memory allocated from the OS for the first thread; can be reused when you start the second thread.

When you run more than one thread concurrently, the process will obviously need more memory; but as old threads end and new ones start; the memory will be recycled for those new threads.

In other words; you aren't seeing a memory leak; just memory usage, so stop worrying about it.

Now, if you are running out of memory, then you are either: a) not terminating old threads cleanly; b) or using more threads than you have memory to support.

If the former; there is usually a fairly obvious reason why threads fail to end cleanly; but I'd need to see the real code.

If the latter; it is usually a design flaw. Most modern systems (4GB min) can support upwards of 100 concurrently threads -- with care, I have had over 3000 concurrent threads running on my 8GB system -- but just because you can have a large number of threads concurrently running; it rarely if ever makes sense to do so. And a design that requires it is usually the wrong design.

Sometimes running out of memory with low(ish) numbers of threads indicates that you are inadvertently duplicating large chunks of memory that you did not intend to.

And finally, there are steps you can take to reduce the memory usage of your threads by careful consideration of what and when you load modules. The default habit of useing modules at the top of the program means that they all get replicated into every thread; regardless of whether those threads need them or not. It's convenient; but can cause excessive memory use.

The bottom line is; if you have a genuine problem with memory usage of your threaded code; and you want to address it; you will have to post the real code, not snippets.

[DNAb]

I can post the code later on tomorrow, that's not really an issue. Eventually I do run out of memory on the system, there is currently 512MB assigned but I can go to 1GB. Threads are killed using a signal which calls threads->exit(). I'm not really sure this is part of the issue though, since again the issue is present in just that small piece of code, and I don't kill threads there, I just detach them.

I guess what you are saying is this is basically just how memory management is in PERL? How else would you deal with a long running application using threads not eating up all the memory?

And, of course, thank you!