Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

understanding devel::leak

by nonnonymousnonk (Novice)
on Oct 04, 2007 at 09:21 UTC ( #642604=perlquestion: print w/replies, xml ) Need Help??

nonnonymousnonk has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,

I've written a fair ammount of perl over the years but now I'm working on a script that needs to run for a week or so. I'm just starting to get to grips with the debugger. My script is consuming large amounts of memory, enough that it will only run for a few hours. Everything is strict;
Devel:leak is showing me things like:

old (0): 0 old (0): 0 old (0): 0 new 0248BF0C : new 0248BFC0 :
What on earth does that mean? How do I use it to track down the problem? Is this a red herring? Is there anything missing that compiling with -DDEBUGGING would add? If so am I stuffed, I'm using Activestate on WinXP? I can't move to a different platform because of the applications that perl is talking to.

Alternatively can you give me any pointers to good guides that will walk me through debugging memory problems from beginner to advanced?

Cheers
nonk

Replies are listed 'Best First'.
Re: understanding devel::leak
by almut (Canon) on Oct 04, 2007 at 10:43 UTC
    Is there anything missing that compiling with -DDEBUGGING would add?

    Yes, if your perl had been built with -DDEBUGGING, Devel::Leak would call its (then available) routine sv_dump() on the things in question, which would produce output similar to what you'd get with the module Devel::Peek. How much that helps, though, will depend on the particular situation...

    Anyhow, most problems with memory leaks in Perl code are related to circular references. Consider the following

    for (1..1000000) { my $h = {}; $h->{myself} = $h; }

    This harmlessly looking piece of code consumes about 130MB (on my machine). Reason is that the anonymous hashes (referenced by $h) are not being freed, because they're holding a reference to themselves, so their reference count doesn't drop to zero when the $h goes out of scope...

      Thanks for clarifying the debugging thing, I had a feeling something must be missing. If anyone has any pointers to getting devel::leak fully working I'd be grateful.

      I don't think I've created any circular references, I've certainly not got any re-entrant code. However, I have loads of anonymous arrays. I did find_cycle and Dump() a few variables at function exit points and found all had ref counts greater than 1. I'll see what I can find with devel::peek.

      edit Oops, Dump() is devel::peek, I was associating it with data::dumper. Ho hum.

      I've just added an undef ....; to the end of all relevent functions to correspond with all the my ......; but I feel like I'm guessing. There must be a better way to get an idea of memory use than watching perfmon.

        I've just added an undef ....; to the end of all relevent functions to correspond with all the my ......;

        Whether undef-ing works, depends on what exactly you undef. In the trivial example above, undef $h at the end of the loop wouldn't help (as that's implicitly happening anyway), while undef %$h, undef $h->{myself} or delete $h->{myself} would help. Point is that you have to break the circle before the data structure becomes unreachable...

        In real life, self-referencing circles are often not immediately evident, as they may come into being indirectly through several data structures, e.g. A references B, which references C, which ... references A — or some such. Your ref counts > 1 might hint at such a situation, unless you have another good explanation for them :)

        Sometimes (if all else fails), it helps to step by step disable parts of the program, until the problem goes away. In this case, look more closely into whatever that last part was, etc. — Without seeing any actual code, it's hard to come up more specific tips.

Re: understanding devel::leak
by Nova (Novice) on Oct 04, 2007 at 15:25 UTC
    Hi nonk ,
    This theme seems to be quite common , memory gets eaten up and is not released .
    The issue is eroding my own confidence in proceeding to develop a perl based tool which I would like to be able to control the release of MEM , when not needed via the best possible code .
    On my NT PC / with Perl running under ActiveState , the following code eats up 92,608 KB MEM . After the undef the MEM usage drops only to 90,572 KB MEM . An awefull lot of Residual MEM is not returned to the O/S until the program exits .
    I know that I am pushing an anonymous array into X , but it really concerns me that undef is not reclaiming the MEM . Can my code be modified to allow the MEM to be reclaimed to the O/S . I have not found a way
    Nova <code> #!perl { my $i=0 ; my @X ; while ( $i < 500000 ) { push @X , $i , $i , $i , $i ; if ($i%10000==0) { printf "Index %-7d %-7d \n\n" , $i , $X$i2 ;} $i++ ; } ; printf "Check MEM usage then Return to undef X" ; <> ; undef @X ; }; printf "Enter Return to quit program" ; <> ; <\code>

      The problem you are having is not a 'true' memory leak, i.e. the memory in fact is being freed. It just isn't being returned to the OS, but rather to Perl's own memory pool. Those two things are often confused. The difference is that if you'd run your while loop another time, it would not allocate another 90MB, but rather reuse the memory freed upon undef-ing the @X.  If it were a memory leak in the sense that the rest of the thread is talking about, it'd eat up more and more memory, without you having a chance to reuse it (without exiting the interpreter).

        Thank you for your responses and I apologize for the format problem in my previous post .
        From the point of view of the O/S , I can see that , once I have done work via my script which requires MEM , a large chunk of that MEM may no longer be available to the O/S for concurrently running apps .
        My perl app is running on WIN32 / ActiveState Perl 5.8.8 . Can I set any variables which would apply the -DPURIFY after Perl has already been compiled , or the binaries intalled .
        Further , while the script is running is there a variable which can be examined which I can use to monitor MEM usage .
        Kind regards, Nova .

      I recently learned you can compile your perl with the parameter -DPURIFY which disables perl's own memory pool. This means whenever you release memory it is visible outside the process. Normally freed memory is just available to perl to do new things with. Perl's own pool may also free parts of itself but that's a separate issue from your structures as visible from within your program. This is just an effect of using a language with its own garbage collection.

      ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊

        Thank you Monks for the pointers.

        After several weeks of bashing my head against this problem I have at last made a breakthrough, of sorts.

        Along my voyage of discovery I've tried:
        Devel::leak - no use as I've no idea how to recompile on win32 / activestate. I downloaded the source and read the readme but it just left me even more confused.
        Devel::Leak::Object - Don't recall why I gave up on that one.
        Devel::Cycle - My data structure is pretty complex so it was entirely plausible that I'd looped somewhere. I thought I wasn't using find_cycle right but it turns out there was nothing there to find.
        Devel::Monitor - plain doesn't work on win32 / activestate.
        Devel::Peek - I sort of understood what this was telling me but not entirely.
        Win32::Perfmon - doesn't work for any of the counters that have instances.

        In the end all I could do was step through with the degugger looking for patterns in where perls private bytes increased. Not easy due to the way perl grabs memory in 4k chunks. In the end I got down to one suspect package function. I wrote a fake version of this that returns realistic data. Using this I have proved that there are absolutely no memory leaks in my code. I'd be quite please with this if only it did not render 6 weeks effort pretty worthless.

        The module causing the problem is Win32::OLE:OPC and the function is SyncRead.

        I'll have a brief look inside the package but I think I reached my limit. It's entirely possible that the problem isn't in that package or even in any of it's perl dependancies but elsewhere in OLE or COM.

        Bit depressing really.
        nonk

Re: understanding devel::leak
by brycen (Monk) on Mar 21, 2009 at 00:16 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://642604]
Approved by lima1
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2020-11-27 00:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?