threads causing memory leak

[holli]

Monks.

Please share your wisdom with me in the question of why this code

    my $rss : shared;
    
    my $thread = threads->create( sub { 
        my $ua = LWP::UserAgent->new;
        my $response = $ua->get( $url );

        $rss = $response->decoded_content
            if $response->is_success;
    } );

is leaking memory like there is no morning. This code is called every 30 seconds, and adds up roughly 20 megs of memory at each call. That of course leads to out of memory crashes pretty quickly.

Making it non threaded, the problem goes away, but then my Win32::GUI gets unresponsive.

Thanks for your time.

Btw, this is on Windows XP, using the latest Strawberry Perl.

[Corion]

You say "the code is called every 30 seconds" - do you mean you create a new thread every 30 seconds? Do you ever ->join that thread? Otherwise, I think you should use async. Personally, I would put the fetched results in a Thread::Queue and retrieve it from there in the main thread, so that all updating of new data happens within the main thread and only the fetching happens asynchronously, using ->dequeue_nb().

As Win32::GUI is event-driven, it might even make sense to have the worker thread send a window message to the main program to tell it to fetch the result.

[holli]

do you mean you create a new thread every 30 seconds?

Yes.

Do you ever ->join that thread?

No, i wait for it to finish.

Otherwise, I think you should use async.

async?

Update:
Using async does not help, same problem. joining the thread leads to:

Free to wrong pool 9f2520 not 3f4e10 at c:/strawberry/perl/vendor/lib/
+Win32/GUI.pm line 2703 during global destruction.

[BrowserUk]

No, i wait for it to finish.

That's exactly what join does. But if you don't join it, it's memory is never returned to the memory pool.

Free to wrong pool 9f2520 not 3f4e10

That's a bug in Win32::GUI.

[BrowserUk]

Here you go. Now tested. It fetches the url every 5 seconds ad nauseum, and no leaks.

#! perl -slw
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use LWP::UserAgent;

sub t {
    my $Q = shift;
    while( $Q->dequeue ) {
        my $ua = LWP::UserAgent->new;
        my $response = $ua->get(
            'http://www.mektek.net/mekmatch/listServersRss.mkz'
        );

        my $rss = $response->decoded_content
            if $response->is_success;

        print $rss if $rss;
        print "done1";
    }
}

my $Q = new Thread::Queue;
my $thread = threads->create( \&t, $Q );

require Win32::GUI;

my $mw = Win32::GUI::Window->new(
    -width => 1000, -height => 800,
    -title => "Test", -name => "Test",
    -onTimer => sub {

        $Q->enqueue( 1 );

    }
);

my $use_up_some_memory = "x" x (1024 * 1024 * 20);

$mw->AddTimer( "T1", 5000 );
$mw->Show;

Win32::GUI::Dialog();
$Q->enqueue( undef );
$thread->join;

[nikosv]

Free to wrong pool

I've been bitten by that as well when using Tk.It is an issue with reentrancy and/or thread safety.You might want to check this node out for more info Perl Tk and threads

You might also want to follow Corion's advise of sending a Window message back to the GUI thread since the call will be serialized in its message queue and thus won't break thread safety.

You might also run the app under Leakdiag and LDGraph which parses the data collected from Leakdiag and presents a nice diagram with the suspicious memory allocations so it might help you pinpointing the leak

[zentara]

In Perl, there is this f*ing ref count thing, which cause complex, non-threadsafe modules, to not be completely destroyed, due to them still wanting to use a variable.

You can google for "perl memory thread safety" and see the gory details. My advice, if you want to play it safe, use a reusable-thread model in your software design. You create threads once, then reuse them, avoiding the need to worry about threads-based memory gains to the perl interpreter. So see Reusable threads demo and OS memory reclamation with threads on linux for examples.

[BrowserUk]

If you post a runnable demo, I'd be happy to investigate.

[holli]

Here you are:

use strict;
use threads;
use warnings;

use Win32::GUI qw();
use LWP::UserAgent;

my $window = Win32::GUI::DialogBox->new( -title => "Test", -name => "T
+est", -onTimer => sub { return poll() } );
my $use_up_some_memory = join( "", "x" x (1024 * 1024 * 20) );

$window->AddTimer( "T1", 20000 );
$window->Show;

Win32::GUI::Dialog();

sub poll
{
    my $rss : shared;
    print "polling\n";
    
    my $thread = threads->create( sub { 
        my $ua = LWP::UserAgent->new;
        my $response = $ua->get( 'http://www.mektek.net/mekmatch/listS
+erversRss.mkz' );

        $rss = $response->decoded_content
            if $response->is_success;
    } );
    
    while ( $thread->is_running ) {
        sleep(1); Win32::GUI::DoEvents;
    }

    return unless $rss;
}

[BrowserUk]

As I suspected it (and Corion called it), you constantly start new threads and never join or detach them, then they just hang around consuming memory once they've completed.

Also, as the last line of the thread sub is $rss, that is returned from the thread, consuming more memory. The fact that you've shared it, as well as returning it means you've consumed even more.

And while I'm critiquing, there are some other anomalies in your code:

1. "x" x (1024 * 1024 * 20) is a single string of 20MB 'x's.

   The join is utterly redundant.

2. return unless $rss;

   Return (nothing) unless $rss has some value.

   Otherwise, just fall off the end of the subroutine and return ???

Try something like this (untested):

#! perl -slw
use strict;
use threads;
use Win32::GUI qw();
use LWP::UserAgent;

sub t {
    my $rss;
    my $ua = LWP::UserAgent->new;
    my $response = $ua->get(
        'http://www.mektek.net/mekmatch/listServersRss.mkz'
    );

    $rss = $response->decoded_content
        if $response->is_success;

    print "done1";
    return $rss;
}

my $window = Win32::GUI::DialogBox->new(
    -title => "Test", -name => "Test", -onTimer => sub { return poll()
+ }
);

my $use_up_some_memory = join( "", "x" x (1024 * 1024 * 20) );

$window->AddTimer( "T1", 5000 );
$window->Show;

Win32::GUI::Dialog();

sub poll
{
    print "polling ... ";

    my $thread = threads->create( \&t );

    while ( $thread->is_running ) {
        Win32::Sleep 10;
        Win32::GUI::DoEvents;
    }
    print " done2";
    return $thread->join;
}

Presumably you want to do something with the data returned from poll?

[Corion]

I think you really need to do a $thread->join at the end of your subroutine.

But instead of polling, I would have the thread send a (Windows) message to the main thread to tell the main thread/window that it can now fetch the data and update things.

[Anonymous Monk]

Might be of interest http://rob.themayfamily.me.uk/perl/win32-gui/win32-gui-threadutils

[Anonymous Monk]

This code is called every 30 seconds, and adds up roughly 20 megs of memory at each call. That of course leads to out of memory crashes pretty quickly.

Um, no, the code you showed only gets called once

[Proclus]

Using threads heavily in Windows GUI applications will lead to problems.

My setup is always built on top of POE and a single worker thread using Thread::Queue. This mechanism has given me quite the flexibility in my projects.

You join the thread just before the application is closed. That seems to be a work around for the crash problems caused by the bugs in the GUI libraries.

In your case, I would have a look at POE-Component-Client-HTTP. Also, you may not need a thread.

[holli]

Thanks all for your efforts. Regarding the OP, I have replaced LWP with HTTP::Async + Win32::GUI::Timer, and now don't need threads at all.

Just to let you know.

[zentara]

ha ha, I knew you would like timers, even if they are not Glib timers. :-)