Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

threads causing memory leak

by holli (Monsignor)
on Sep 11, 2010 at 12:51 UTC ( #859744=perlquestion: print w/ replies, xml ) Need Help??
holli has asked for the wisdom of the Perl Monks concerning the following question:

Monks.

Please share your wisdom with me in the question of why this code
my $rss : shared; my $thread = threads->create( sub { my $ua = LWP::UserAgent->new; my $response = $ua->get( $url ); $rss = $response->decoded_content if $response->is_success; } );
is leaking memory like there is no morning. This code is called every 30 seconds, and adds up roughly 20 megs of memory at each call. That of course leads to out of memory crashes pretty quickly.

Making it non threaded, the problem goes away, but then my Win32::GUI gets unresponsive.

Thanks for your time.

Btw, this is on Windows XP, using the latest Strawberry Perl.


holli

You can lead your users to water, but alas, you cannot drown them.

Comment on threads causing memory leak
Download Code
Re: threads causing memory leak
by Anonymous Monk on Sep 11, 2010 at 13:09 UTC
    This code is called every 30 seconds, and adds up roughly 20 megs of memory at each call. That of course leads to out of memory crashes pretty quickly.

    Um, no, the code you showed only gets called once

Re: threads causing memory leak
by BrowserUk (Pope) on Sep 11, 2010 at 13:11 UTC

    If you post a runnable demo, I'd be happy to investigate.

      Here you are:
      use strict; use threads; use warnings; use Win32::GUI qw(); use LWP::UserAgent; my $window = Win32::GUI::DialogBox->new( -title => "Test", -name => "T +est", -onTimer => sub { return poll() } ); my $use_up_some_memory = join( "", "x" x (1024 * 1024 * 20) ); $window->AddTimer( "T1", 20000 ); $window->Show; Win32::GUI::Dialog(); sub poll { my $rss : shared; print "polling\n"; my $thread = threads->create( sub { my $ua = LWP::UserAgent->new; my $response = $ua->get( 'http://www.mektek.net/mekmatch/listS +erversRss.mkz' ); $rss = $response->decoded_content if $response->is_success; } ); while ( $thread->is_running ) { sleep(1); Win32::GUI::DoEvents; } return unless $rss; }


      holli

      You can lead your users to water, but alas, you cannot drown them.

        I think you really need to do a $thread->join at the end of your subroutine.

        But instead of polling, I would have the thread send a (Windows) message to the main thread to tell the main thread/window that it can now fetch the data and update things.

        As I suspected it (and Corion called it), you constantly start new threads and never join or detach them, then they just hang around consuming memory once they've completed.

        Also, as the last line of the thread sub is $rss, that is returned from the thread, consuming more memory. The fact that you've shared it, as well as returning it means you've consumed even more.

        And while I'm critiquing, there are some other anomalies in your code:

        1. "x" x (1024 * 1024 * 20) is a single string of 20MB 'x's.

          The join is utterly redundant.

        2. return unless $rss;

          Return (nothing) unless $rss has some value.

          Otherwise, just fall off the end of the subroutine and return ???

        Try something like this (untested):

        #! perl -slw use strict; use threads; use Win32::GUI qw(); use LWP::UserAgent; sub t { my $rss; my $ua = LWP::UserAgent->new; my $response = $ua->get( 'http://www.mektek.net/mekmatch/listServersRss.mkz' ); $rss = $response->decoded_content if $response->is_success; print "done1"; return $rss; } my $window = Win32::GUI::DialogBox->new( -title => "Test", -name => "Test", -onTimer => sub { return poll() + } ); my $use_up_some_memory = join( "", "x" x (1024 * 1024 * 20) ); $window->AddTimer( "T1", 5000 ); $window->Show; Win32::GUI::Dialog(); sub poll { print "polling ... "; my $thread = threads->create( \&t ); while ( $thread->is_running ) { Win32::Sleep 10; Win32::GUI::DoEvents; } print " done2"; return $thread->join; }

        Presumably you want to do something with the data returned from poll?


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: threads causing memory leak
by Corion (Pope) on Sep 11, 2010 at 13:11 UTC

    You say "the code is called every 30 seconds" - do you mean you create a new thread every 30 seconds? Do you ever ->join that thread? Otherwise, I think you should use async. Personally, I would put the fetched results in a Thread::Queue and retrieve it from there in the main thread, so that all updating of new data happens within the main thread and only the fetching happens asynchronously, using ->dequeue_nb().

    As Win32::GUI is event-driven, it might even make sense to have the worker thread send a window message to the main program to tell it to fetch the result.

      do you mean you create a new thread every 30 seconds?
      Yes.
      Do you ever ->join that thread?
      No, i wait for it to finish.
      Otherwise, I think you should use async.
      async?

      Update:
      Using async does not help, same problem. joining the thread leads to:
      Free to wrong pool 9f2520 not 3f4e10 at c:/strawberry/perl/vendor/lib/ +Win32/GUI.pm line 2703 during global destruction.


      holli

      You can lead your users to water, but alas, you cannot drown them.
        No, i wait for it to finish.

        That's exactly what join does. But if you don't join it, it's memory is never returned to the memory pool.

        Free to wrong pool 9f2520 not 3f4e10

        That's a bug in Win32::GUI.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: threads causing memory leak
by Anonymous Monk on Sep 11, 2010 at 14:07 UTC
Re: threads causing memory leak
by nikosv (Hermit) on Sep 11, 2010 at 17:00 UTC

    Free to wrong pool

    I've been bitten by that as well when using Tk.It is an issue with reentrancy and/or thread safety.You might want to check this node out for more info Perl Tk and threads

    You might also want to follow Corion's advise of sending a Window message back to the GUI thread since the call will be serialized in its message queue and thus won't break thread safety.

    You might also run the app under Leakdiag and LDGraph which parses the data collected from Leakdiag and presents a nice diagram with the suspicious memory allocations so it might help you pinpointing the leak

Re: threads causing memory leak
by zentara (Archbishop) on Sep 11, 2010 at 19:43 UTC
    In Perl, there is this f*ing ref count thing, which cause complex, non-threadsafe modules, to not be completely destroyed, due to them still wanting to use a variable.

    You can google for "perl memory thread safety" and see the gory details. My advice, if you want to play it safe, use a reusable-thread model in your software design. You create threads once, then reuse them, avoiding the need to worry about threads-based memory gains to the perl interpreter. So see Reusable threads demo and OS memory reclamation with threads on linux for examples.


    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
Re: threads causing memory leak (No leak!)
by BrowserUk (Pope) on Sep 11, 2010 at 20:20 UTC

    Here you go. Now tested. It fetches the url every 5 seconds ad nauseum, and no leaks.

    #! perl -slw use strict; use threads; use threads::shared; use Thread::Queue; use LWP::UserAgent; sub t { my $Q = shift; while( $Q->dequeue ) { my $ua = LWP::UserAgent->new; my $response = $ua->get( 'http://www.mektek.net/mekmatch/listServersRss.mkz' ); my $rss = $response->decoded_content if $response->is_success; print $rss if $rss; print "done1"; } } my $Q = new Thread::Queue; my $thread = threads->create( \&t, $Q ); require Win32::GUI; my $mw = Win32::GUI::Window->new( -width => 1000, -height => 800, -title => "Test", -name => "Test", -onTimer => sub { $Q->enqueue( 1 ); } ); my $use_up_some_memory = "x" x (1024 * 1024 * 20); $mw->AddTimer( "T1", 5000 ); $mw->Show; Win32::GUI::Dialog(); $Q->enqueue( undef ); $thread->join;

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: threads causing memory leak
by Proclus (Beadle) on Sep 12, 2010 at 11:44 UTC
    Using threads heavily in Windows GUI applications will lead to problems.

    My setup is always built on top of POE and a single worker thread using Thread::Queue. This mechanism has given me quite the flexibility in my projects.

    You join the thread just before the application is closed. That seems to be a work around for the crash problems caused by the bugs in the GUI libraries.

    In your case, I would have a look at POE-Component-Client-HTTP. Also, you may not need a thread.


    "Geneva will not marry you no matter how much you love Larry Wall." Geneva Wall.
Re: threads causing memory leak
by holli (Monsignor) on Sep 24, 2010 at 15:24 UTC
    Thanks all for your efforts. Regarding the OP, I have replaced LWP with HTTP::Async + Win32::GUI::Timer, and now don't need threads at all.

    Just to let you know.


    holli

    You can lead your users to water, but alas, you cannot drown them.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://859744]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (10)
As of 2014-07-22 07:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (106 votes), past polls