Strange memory growth?

[meetraz]

Can anybody explain why this code doesn't do what I think it should?

use strict;
sleep 5;

if (1) { my $foo = "perl" x 1000000;  sleep 5; }
if (1) { my $foo = "perl" x 1000000;  sleep 5; }
if (1) { my $foo = "perl" x 1000000;  sleep 5; }
if (1) { my $foo = "perl" x 1000000;  sleep 5; }
if (1) { my $foo = "perl" x 1000000;  sleep 5; }
if (1) { my $foo = "perl" x 1000000;  sleep 5; }

I always thought that if you declare a lexical variable (such as my $foo) inside a block, that it gets freed / garbage-collected at the end of the block.

My understanding is that while perl may not return allocated memory back to the OS, it will at least reuse earlier freed memory, rather than allocating more from the OS.

However, the code above will grow by 8M every 5 seconds. If you include the if-loop enough times, it will consume all available memory (at least on Win2k, perl 5.6.1 and 5.8.0)

[liz]

And with 5.8.2 on Linux as well.

Oddly enough, this doesn't happen if you do:

foreach 1..6 { my $foo = "perl" x 1000000;  sleep 5; }

So, it would seem that Perl doesn't know to reclaim if it is a different scope.

Surprisingly, the following also "loses" memory:

my $foo;
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }

Uses about 28 Mb at the end (the original uses 48 Mb).

Maybe something for 5.8.3 to look at ;-)

Liz

[meetraz]

I wonder why having the my inside the block would cause it to use more memory than having it outside?

And even without the my, it still uses 28M:

sleep 5;

if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }
if (1) { $foo = "perl" x 1000000;  sleep 5; }

[Anonymous Monk]

I can't be sure, but it may have something to do with those if (1) {} blocks getting optimized away.

[hardburn]

It still get's put inside a block:

$ perl -MO=Deparse -e 'if(1) { $foo = "perl" x 1000000; sleep 5; }'
do {
    $foo = 'perl' x 1000000;
    sleep 5
};

[welchavw]

http://nntp.x.perl.org/group/perl.perl5.porters/51374 is interesting and relevant, I think.

,welchavw

[hardburn]

Ahh, now I see. There is the possibility that the if block is inside a loop, so perl will keep the lexical around so we don't have to recreate it on the next run through the loop.

So if this is really a problem, you could always do:

while($condition) {
    # do something

    $condition = 0;  # Make sure condition is false at the end
}

Please, only if you've anaylized the program and you really are in danger of running out of memory (including swap space) from the conditionals you're doing. Anything else is a micro-optimization.

[liz]

Please, only if you've anaylized the program and you really are in danger of running out of memory (including swap space) from the conditionals you're doing. Anything else is a micro-optimization.

Possibly. But please remember when Perl is used in a mod_perl environment with a prefork MPM, you want to keep as much memory shared between children. This behaviour of Perl is very counter-productive in that environment: I'd rather take a little CPU hit for the overhead of creating the lexical again, and not unsharing any further memory, than have the current situation.

More generally, I think the current behaviour of Perl to use more memory rather than more CPU, becomes more and more counter-productive as the speed of accessing memory is growing slower than the speed of the CPU with each new generation of systems.

Liz

[meetraz]

Are you sure?... see my second post above. Even when I use the *same* *package* var, there is still unexplained memory growth. Your point about keeping the lexical around to avoid re-creating it next time through the loop doesn't seem applicable. In that case, I'm not using a lexical, nor am I creating a variable more than once.

Update: Here's an even simpler example:

$foo = 'perl' x 1000000; sleep 5;
$foo = 'perl' x 1000000; sleep 5;
$foo = 'perl' x 1000000; sleep 5;
$foo = 'perl' x 1000000; sleep 5;
$foo = 'perl' x 1000000; sleep 5;
$foo = 'perl' x 1000000; sleep 5;

Here I'm assigning the same value to the same variable over and over, why is memory use increasing? It seems too simple to be a bug, but still seems hard to explain.

[tilly]

As stated by others, this is known behaviour.

For a related bug, see Unusual Closure Behaviour.

[fletcher_the_dog]

Check out this code. It looks like in a subroutine the clean up of lexical variables in a "if" statement is not done until after the sub exits:

use strict;
package foo;
our $counter = 0;

sub new {
  my $class = shift;
  bless {id=>$counter++},$class;
}

sub DESTROY{
  my $self = shift;
  print "DESTROYING $self->{id}\n";
}

package main;

test();
print "Back in main\n";
sub test{
  print "Entering test\n";
  if (1) {
    my $foo = foo->new();
  }
  if (1) {
    my $foo = foo->new();
  }
  if (1) {
    my $foo = foo->new();
  }
  print "Leaving test\n";
}

__OUTPUT__
Entering test
Leaving test
DESTROYING 0
DESTROYING 1
DESTROYING 2
Back in Main

UPDATE The point of showing this code, is that maybe it is not a memory leak, but merely a delay in clean up

[OverlordQ]

Might look at

perldoc perlsub

namely the

Persistent Private Variables

section, dunno if anything there applies to this situation.

[Taulmarill]

the if block is no lexical block like the blocks you declare for while, for, foreach, etc. so the $foo scalar lives in the whole script.

[fletcher_the_dog]

Um, that is not true, and if it was then it is more likely that the memory growth would not happen because $foo would be overwritten each time.

use strict;
if (1) {
  my $foo = "hi";
}
print $foo."\n";

__OUTPUT__
Global symbol "$foo" requires explicit package name at scope.pl line 5
+.
Execution of scope.pl aborted due to compilation errors.

[Taulmarill]

me stupid, you are absolutely right fletcher.

[mrpeabody]

Wrong. Lexical variables declared inside an if BLOCK are not visible outside that block (and its associated else/elsif BLOCK). perlsub:

The "my" operator declares the listed variables to be lexically confined to the enclosing block, conditional ("if/unless/elsif/else"), loop ("for/foreach/while/until/continue"), subroutine, "eval", or "do/require/use"'d file.

As can be seen:

502 $ perl -we'use strict;if(1){my $foo="foo"};print $foo'
Global symbol "$foo" requires explicit package name at -e line 1.
Execution of -e aborted due to compilation errors.

The scoping rules work as they should, but the memory usage could definitely be more conservative.

edit: beaten

[pg]

This is still a bug. Try this:

use strict;
use warnings;

if (1) {
    my $a = 100;
}
print $a;

If you are not allowed to ref that $a any more, once exit that if block, there is absolutely no point not to garbage collect it.

[grinder]

there is absolutely no point not to garbage collect it

Yes there is. It might be cheaper, (read: Lazier) to not garbage collect it at all, and just let the program fall off the end of the world. Just because you can, doesn't mean it's worth spending the effort to do so.

It was only with the advent of mod_perl and similar long-running persistent perl processes that any serious effort started to be devoted to keeping track of perl's resource usage. It used to be that it was cheaper not to do anything, because in a few milliseconds time, the pages used by the process would be reclaimed by the OS when program ended anyway.

And given perl's propensity for trading off memory for speed, it doesn't really surprise me that we see the behaviour demonstrated by the OP.

[mce]

[ambrus]

Isn't that because ("perl" x 1000000) is a constant so it's subject to constant optimization, it's calculated at complile time?