undefining hashes to free memory

[tigervamp]

I tracked down an apparent memory leak in a program I wrote to the way perl seems to deal with hashes. The program uses some fairly large hashes that need to be completely redefined on a regular basis. I had been using "my" within a block scope thinking that the memory allocated for the hash would be freed (at least for use by perl for later) when the end of the block was reached. This does not seem to be exactly the case. In the first (simplified) example:

{
my %hash;
for my $i (1 .. 500000) {
        $hash{$i}=1;
}
}
sleep 1;

{
my %hash;
for my $i (1 .. 500000) {
        $hash{$i}=1;
}
}
sleep 1;


{
my %hash;
for my $i (1 .. 500000) {
        $hash{$i}=1;
}
}
sleep 1;

...

The amount of memory needed for the hash is 36MB (on my system). After the first block completes though, every other block increases the size of the memory being used by the program by a couple of MBs. However, when I add "undef" statements inside the blocks like so:

{
my %hash;
for my $i (1 .. 500000) {
        $hash{$i}=1;
}
undef %hash;
}
sleep 1;

...

this does not occur, the resident memory stays constant at 36 MB. My question is: Why is extra memory being used in the first example but not the second and is the latter scenerio the best way to handle this? I have tried this on Linux and SCO UNIX with perl 5.8 and 5.005_03 respectively, I think this is a perl thing. Any insight is greatly appreciated.

Thanks,

tigervamp

[dmitri]

A while back, I searched perlmonks and googled to get a better understanding of how perl manages memory. These are the links I've collected:

PM: Strange memory growth?

PM: Nested Calls, Recursion and MEMORY

nntp.perl.org - perl.perl5.porters (51374)

When perl is not quite fast enough

PM: memory usage/leaks

PM: memory usage woes

PM: Garbage collection of 'my' variables

PM: Tracking Memory Leaks

Re: Is perlbug broken?

[Anonymous Monk]

thank you

[tachyon]

In addition to the above you can't always rely on Perl to do the right thing with regards to memory return. In our experience perl never really gives back memory to the OS until absolutely forced to. This is referred to as lazy garbage collection by p5p and semi-non-existent garbage collection by some more cynical punters. Also undeffing large hashes can take a very long time on some systems due to a known malloc bug. You may find value in effectively making the reused hashes global in some circumstances:

my %hash;

...

sub process {
    %hash = ();  # empty hash but not using undef
    # do stuff with hash
}

As always YMMV.

[Anonymous Monk]

Thanks for clarification. As far as giving memory back to the OS, I don't think it's unreasonable not to, but I also think it's reasonable to expect efficient reuse of memory from out of scope variables, maybe that's too much to expect ;) I did test your approach before I posted and it seemed to work but is a little messy in my situtation, I also know about the malloc bug and it doesn't seem to be an issue on my systems, but thanks for the informative response.

tigervamp

[perrin]

Perl holds onto the memory used by lexicals in case you use them again. It's a performance optimization. Undef'ing them releases the memory and lets perl use it for something else.

[Joost]

An interesting optimization. Exactly what gets kept, then? I'm guessing it's the probably the array to hold the key references.

/me notes that this will only become a problem if you have lots of different lexicals that hold a lot of data - one after the other - which is probably not a common case, and easily avoided by undef'ing them as demonstrated above

[Anonymous Monk]

How could I possibly reuse it? The hash has gone out of scope, there are no external references to the hash or it's elements; I don't see any way of addressing any of it later.

[perrin]

If you run the same sub again, which uses the same lexical again, perl reuse the same memory for it.

[waswas-fng]

The memory used and released back into the processes' pool may be getting slightly fragmented in the first circumstance. you can't use the same block of memory for the next task if there is still a portion in the middle that is still used, you have to request more. In the second method you may be freeing the entire block of memory to the pool so you do not notice the (probable) allocation.

[eserte]

Perl may return memory to the OS by calling free() internally, but the OS may not be able to mark the memory really as free. Memory is usually organized in a linear heap. Suppose you allocate a big chunk of memory, and then a very small chunk of memory. Now the big chunk is freed, but the small chunk (which is allocated after the big one) is not. Memory can't be compacted by moving chunks around (think of pointers), so the consequence is that you have a big chunk of unused memory. Consecutive mallocs would use part of the big chunk, but you stay with a maximum peak.

[zentara]

I run into that problem alot doing Tk programming, since all Tk widgets are hashes of one sort or another. The undef, delete, or destroying of these hashes does not free the memory, and if you design that way, you will get applications which just keep growing in size. It's not too noticeable on one-shot quick GUIs, but on long-running programs, it can really become noticable.

In Tk, the only solution is to make your objects only ONCE and then reuse them. The previously mentioned solution about making the hash a global, then emptying it and reusing it, sounds about the right approach to me. Also watch out for auto-vivication of hashes, which causes them to "leak", read Autovivification with hash of hashes for example. It can be hard to effectively delete or undef an hash.