Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re^2: Tracking down memory leaks

by Hena (Friar)
on Apr 13, 2005 at 12:47 UTC ( #447343=note: print w/replies, xml ) Need Help??


in reply to Re: Tracking down memory leaks
in thread Tracking down memory leaks

A perl program can use more memory after running a while even if nothing is wrong. For example, if you load a 10MB file into a scalar, that scalar will hang onto that memory, even if it goes out of scope.
Shouldn't these situations be handled by garbage collection. If a scalar (or array/hash) gets out of scope (eg. subroutines internal variables) they should be freed when memory is needed before asking memory from system?

AFAIK memory asked from system won't shrink.

Replies are listed 'Best First'.
Re^3: Tracking down memory leaks
by Joost (Canon) on Apr 13, 2005 at 12:59 UTC
    Shouldn't these situations be handled by garbage collection.
    Maybe, but mostly they aren't, for performance reasons:
    sub bla { my $arg = shift; my $big_string = $arg x 1000; }

    Perl will in general keep the memory for $big_string allocated and reserved, because then it doesn't need to allocate the memory again next time the sub is called.

    Explicitly undef()ing or resizing variables before they go out of scope sometimes helps, though - on some systems, it might even free the memory back to the system.

    Usually, you don't need to do this, exactly because the memory gets reused anyway. If your program grows a lot, it's more likely you're using an inefficient algorithm, or you're creating circular references somewhere.

      So basicly if one has a large variable, which should be removed by scoping (not counting the speed usage but memory in should word), one should do manually. But if the large variable is within sub and that sub is used again (eg from a loop structure), then all the variables within it is automatically reused.
        That's how I understand it, yes. Lexicals are re-used when iterating though a loop / subroutine note: recursive subroutine calls create a "stack" of lexicals - for obvious reasons - lookup goto &sub if you want to do tail recursion.

        Ofcourse, if your program is structured well enough, you usually don't have to worry about it - what I mean is, if your variables go out of scope, you usually are going out of a sub or loop in which they will be used again.

        The usual exception is some code that runs in stages, and you want to free as much memory as possible after you're done with one stage and enter the next, for instance, some code that needs to preprocess a lot of data in memory, write it out to a file and then go through the resulting file line by line - or something like that.

      Note that in your example, Perl will keep the memory for $big_string twice:
      #!/usr/bin/perl use strict; use warnings; use Readonly; Readonly my $size => 10 * 1024 ** 2; # 10 Mb. sub show_mem { system "grep ^VmSize /proc/$$/status"; } sub gimme_big { my $size = shift; my $var = 'x' x $size; } show_mem; gimme_big $size; show_mem; __END__ VmSize: 3464 kB VmSize: 23952 kB
      For a 10Mb string, Perl allocates about 20Mb memory.

      undefing the variable makes Perl allocate about 10Mb less:

      #!/usr/bin/perl use strict; use warnings; use Readonly; Readonly my $size => 10 * 1024 ** 2; # 10 Mb. sub show_mem { system "grep ^VmSize /proc/$$/status"; } sub gimme_big { my $size = shift; my $var = 'x' x $size; undef $var; } show_mem; gimme_big $size; show_mem; __END__ VmSize: 3472 kB VmSize: 13716 kB
      So, how do we get rid of the extra 10Mb? By a careful use of string eval:
      #!/usr/bin/perl use strict; use warnings; use Readonly; Readonly my $size => 10 * 1024 ** 2; # 10 Mb. sub show_mem { system "grep ^VmSize /proc/$$/status"; } sub gimme_big { my $size = shift; my $var = eval "'x' x $size"; undef $var; } show_mem; gimme_big $size; show_mem; __END__ VmSize: 3468 kB VmSize: 3468 kB
Re^3: Tracking down memory leaks
by scain (Curate) on Apr 13, 2005 at 12:58 UTC
    That's what I was thinking too. The fact that memory usage slowly grows until the OS kills it is not good. The files I am processing have fairly short lines and it processes the files line by line in a loop. All of the variables inside that loop should be "reusing" the same space, right? Otherwise, what is the point of scoping at all.

    Scott
    Project coordinator of the Generic Model Organism Database Project

      All of the variables in that loop will retain their memory between calls and reuse it. However, if you leave that loop, they will still retain that memory. Also, if you load one of them once with 10MB of data, it will stay at that size until perl exits unless you explicitly undef it when you're done with it.
Re^3: Tracking down memory leaks
by perrin (Chancellor) on Apr 13, 2005 at 13:35 UTC
    No, the memory is not freed. Perl keeps it as an optimization since you would need to allocate it again the next time this chunk of code runs. Of course that doesn't help you any if that code doesn't run again in your program... We have discussed this at length on the mod_perl list and it is covered in the mod_perl docs.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://447343]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (7)
As of 2020-07-07 13:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?