Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Tracking Memory Leaks

by Hrunting (Pilgrim)
on Aug 14, 2001 at 22:02 UTC ( #104846=perlquestion: print w/ replies, xml ) Need Help??
Hrunting has asked for the wisdom of the Perl Monks concerning the following question:

I've got a long-running perl process. I know that perl will use memory from deleted variables, so after a while I expect perl's memory usage to flat line, but it keeps rising and rising. Two questions:

Why?

What is the best way to quickly see what variables/references are using how much memory, a sort of memory snapshot if you will?

Comment on Tracking Memory Leaks
Re: Tracking Memory Leaks
by dragonchild (Archbishop) on Aug 14, 2001 at 22:19 UTC
    Perl's memory should flat-line ... if you programmed it right. Some things to check:
    1. Are you continuously adding elements to a hash or array? For example, if you keep doing something like $foo[$i++] = $bar;, you keep telling Perl to increase the size of @foo.
    2. Do you have any circular references? They are a cause of true memory leaks.
    3. Are you scoping things as much as possible? Do you have any large global hashes/arrays that you could scope a little smaller?
    4. Are you constantly loading new classes/modules? I was on a project for a OO application that would dynamically load the classes it needed as it needed to. Every time that happened, the memory usage would increase, but only a fixed amount.
    5. Are you fooling around with deleting from %INC and @INC? I'm not sure what that would do, but it probably won't actually remove the initial require from RAM.

    ------
    /me wants to be the brightest bulb in the chandelier!

    Vote paco for President!

      Heh, well, we do all those things, but they should be controlled.

      1. Yes, in various places, but they should all be limited in one way or another (either they go out of scope at various points or they only have a limited number of elements in them).
      2. We do have one module that uses circular references, but using Weakref, that shouldn't be a problem.
      3. Yes, scoping on the project is very tight.
      4. This is the thing that I'm worried about. We're constantly loading new classes, dymanically created from database information. Now, theoretically, we have a finite number of classes, so the memory usage from loading these things should be finite itself (and that limit should be hit rather quickly). However, I'm not positive this is the case. Anyway know much about the finer points of this?
      5. No. Thanks for the help.
        To reply to a few points:
        1. Be very careful. Simply because something is out of scope does not mean that it's been garbage-collected. Directly from the Camel (3rd ed., p.223):

          Lexical variables don't get automatically garbage collected just because their scope has exited; they wait to get recycled until they're no longer used, which is muych more important. To create private variables ...

          I would suspect that this is a minor problem, but probably not the whole issue.

        2. As for the constantly loaded classes ... I'm going to guess that you're creating the classes using some sort of AUTOLOAD(), eval, or combination of the two. If that's the case, these classes are probably taking quite a chunk of memory. Now, by doing it this way, you're taking away any possible compile-time optimizations that the compiler could do. In addition, I think that once you use AUTOLOAD() once, the compiler will allocate additional memory every time, just as if you used $` or $' in regexps.

          I think that this is the culprit. I would suspect that it's not that you're loading 10^12 classes, but that each class is taking a bigger chunk than you realize.

        ------
        /me wants to be the brightest bulb in the chandelier!

        Vote paco for President!

        We're constantly loading new classes, dymanically created from database information. Now, theoretically, we have a finite number of classes, so the memory usage from loading these things should be finite itself (and that limit should be hit rather quickly). However, I'm not positive this is the case. Anyway know much about the finer points of this?

        Are you cleaning up the symbol tables after you're done using these new classes, or are they being re-used? We found in Class::Prototyped that each new class takes up around 1.5-2K of memory; more (of course) with methods. You can clean up the symbol table when you're done with code like this (assumes $package does not contain '::'):

        no strict 'refs'; foreach my $key ( keys %{"$package\::"} ) { delete ${"$package\::"}{$key}; } # this only works because we're not a multi-level package: delete( $main::{"$package\::"} );
Re: Tracking Memory Leaks
by Cirollo (Friar) on Aug 14, 2001 at 22:20 UTC
    Try running your script in the perl debugger. You can pretty-print all the variables in a package along with their values using the V and X commands if you want to know what variables you are currently using.

    See the section on "Debugging Perl memory usage" in perlman:perldebug.

Re: Tracking Memory Leaks
by koolade (Pilgrim) on Aug 14, 2001 at 22:40 UTC

    Data::Dumper is a good tool to help track down circular references. If you see something like:

    $VAR1 = \$VAR1; or $VAR1 = [ $VAR1 ]; or $VAR1 = { key => $VAR1 };

    Then you know you have something to fix.

    Also, I've experienced a few problems w/ memory leaks in perl 5.6.0 and linux. e.g.:

    eval "use Any::Module";

    Leaks memory for me, and doesn't appear to with 5.6.1. But I haven't done enough testing to be definitive about that. Someone else might be able to tell you more about this.

Re: Tracking Memory Leaks
by perrin (Chancellor) on Aug 15, 2001 at 04:23 UTC
    You should keep in mind that when lexical variables go out of scope, the memory used by them is NOT reclaimed by Perl. This is an optimization, assuming you will use the lexicals again. If you don't want this behavior, you can explicitly set the variables to undef when you're done with them, which will free their memory.

    Are you doing anything with subroutine references? It's possible to chew up RAM by using things like Error.pm's try/catch syntax if you aren't careful to avoid accidental closures.

    You might want to check out the mod_perl guide for additional information on Perl memory management.

      perrin++

      Just to be clear, the memory used directly by the lexical is not freed but if, for example, the lexical contains the last reference to something, then that something will be destroyed and its memory freed when the lexical's scope is left (or when the last reference to the lexical is destroyed). So this is mostly a problem when dealing with long strings rather than objects.

      And the bug with closures never being freed has been found but I don't think any Perls have been released that contain this fix.

              - tye (but my friends call me "Tye")
        Just to be clear, the memory used directly by the lexical is not freed but if, for example, the lexical contains the last reference to something, then that something will be destroyed and its memory freed when the lexical's scope is left (or when the last reference to the lexical is destroyed). So this is mostly a problem when dealing with long strings rather than objects.

        No, I don't think that's correct. The memory will not be freed, if by freed you mean "available for use by other variables in my program". It will be re-used if you use that lexical again. See this post from Doug MacEachern for more info on this.

        I'm not sure the closures thing I was talking about is really a bug; it's just a side-effect of nested subs and closures. If you do this, it will leak like crazy:

        my $foo; sub bar { sub baz { $foo++; } }

        It's creating a new private copy of $foo for baz() every time. Using Error.pm can sometimes lead people to do this without realizing it, when they use nested try/catch blocks.

Re: Tracking Memory Leaks
by Nitsuj (Hermit) on Aug 15, 2001 at 17:39 UTC
    Important also is "which perl interpretter" and "which OS" different implementations and operating systems will treat garbage collection differently. Just because the perl interpretter free()'d it doesn't mean that the operating system caught that yet... but of course, if perl was maintaining its own heap, that wouldn't matter, but the interpretter would keep the memory until terminated.

    IE, I know that when I run perl, it will behave like this, but what about active perl... or the interpretter that comes with matlab.

    Just Another Perl Backpacker
Re: Tracking Memory Leaks
by Starky (Chaplain) on Aug 16, 2001 at 01:19 UTC
    Take a look at Nick Ing-Simmons' Devel::Leak.

    Hope this helps!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://104846]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (19)
As of 2014-10-30 17:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (208 votes), past polls