Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Strange memory growth?

by meetraz (Hermit)
on Nov 07, 2003 at 20:08 UTC ( #305427=perlquestion: print w/ replies, xml ) Need Help??
meetraz has asked for the wisdom of the Perl Monks concerning the following question:

Can anybody explain why this code doesn't do what I think it should?

use strict; sleep 5; if (1) { my $foo = "perl" x 1000000; sleep 5; } if (1) { my $foo = "perl" x 1000000; sleep 5; } if (1) { my $foo = "perl" x 1000000; sleep 5; } if (1) { my $foo = "perl" x 1000000; sleep 5; } if (1) { my $foo = "perl" x 1000000; sleep 5; } if (1) { my $foo = "perl" x 1000000; sleep 5; }
I always thought that if you declare a lexical variable (such as my $foo) inside a block, that it gets freed / garbage-collected at the end of the block.

My understanding is that while perl may not return allocated memory back to the OS, it will at least reuse earlier freed memory, rather than allocating more from the OS.

However, the code above will grow by 8M every 5 seconds. If you include the if-loop enough times, it will consume all available memory (at least on Win2k, perl 5.6.1 and 5.8.0)

Comment on Strange memory growth?
Download Code
Re: Strange memory growth?
by liz (Monsignor) on Nov 07, 2003 at 20:42 UTC
    And with 5.8.2 on Linux as well.

    Oddly enough, this doesn't happen if you do:

    foreach 1..6 { my $foo = "perl" x 1000000; sleep 5; }

    So, it would seem that Perl doesn't know to reclaim if it is a different scope.

    Surprisingly, the following also "loses" memory:

    my $foo; if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; }
    Uses about 28 Mb at the end (the original uses 48 Mb).

    Maybe something for 5.8.3 to look at ;-)

    Liz

      I wonder why having the my inside the block would cause it to use more memory than having it outside?

      And even without the my, it still uses 28M:

      sleep 5; if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; } if (1) { $foo = "perl" x 1000000; sleep 5; }
      I can't be sure, but it may have something to do with those if (1) {} blocks getting optimized away.

        It still get's put inside a block:

        $ perl -MO=Deparse -e 'if(1) { $foo = "perl" x 1000000; sleep 5; }' do { $foo = 'perl' x 1000000; sleep 5 };

        ----
        I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
        -- Schemer

        : () { :|:& };:

        Note: All code is untested, unless otherwise stated

Re: Strange memory growth?
by OverlordQ (Hermit) on Nov 07, 2003 at 20:45 UTC
    Might look at
    perldoc perlsub
    namely the
    Persistent Private Variables
    section, dunno if anything there applies to this situation.
Re: Strange memory growth?
by Taulmarill (Deacon) on Nov 07, 2003 at 20:54 UTC
    the if block is no lexical block like the blocks you declare for while, for, foreach, etc. so the $foo scalar lives in the whole script.
      Um, that is not true, and if it was then it is more likely that the memory growth would not happen because $foo would be overwritten each time.
      use strict; if (1) { my $foo = "hi"; } print $foo."\n"; __OUTPUT__ Global symbol "$foo" requires explicit package name at scope.pl line 5 +. Execution of scope.pl aborted due to compilation errors.
        me stupid, you are absolutely right fletcher.

      This is still a bug. Try this:

      use strict; use warnings; if (1) { my $a = 100; } print $a;

      If you are not allowed to ref that $a any more, once exit that if block, there is absolutely no point not to garbage collect it.

        there is absolutely no point not to garbage collect it

        Yes there is. It might be cheaper, (read: Lazier) to not garbage collect it at all, and just let the program fall off the end of the world. Just because you can, doesn't mean it's worth spending the effort to do so.

        It was only with the advent of mod_perl and similar long-running persistent perl processes that any serious effort started to be devoted to keeping track of perl's resource usage. It used to be that it was cheaper not to do anything, because in a few milliseconds time, the pages used by the process would be reclaimed by the OS when program ended anyway.

        And given perl's propensity for trading off memory for speed, it doesn't really surprise me that we see the behaviour demonstrated by the OP.

      Wrong. Lexical variables declared inside an if BLOCK are not visible outside that block (and its associated else/elsif BLOCK). perlsub:
      The "my" operator declares the listed variables to be lexically confined to the enclosing block, conditional ("if/unless/elsif/else"), loop ("for/foreach/while/until/continue"), subroutine, "eval", or "do/require/use"'d file.
      As can be seen:
      502 $ perl -we'use strict;if(1){my $foo="foo"};print $foo' Global symbol "$foo" requires explicit package name at -e line 1. Execution of -e aborted due to compilation errors.
      The scoping rules work as they should, but the memory usage could definitely be more conservative.

      edit: beaten

Re: Strange memory growth?
by fletcher_the_dog (Friar) on Nov 07, 2003 at 21:31 UTC
    Check out this code. It looks like in a subroutine the clean up of lexical variables in a "if" statement is not done until after the sub exits:
    use strict; package foo; our $counter = 0; sub new { my $class = shift; bless {id=>$counter++},$class; } sub DESTROY{ my $self = shift; print "DESTROYING $self->{id}\n"; } package main; test(); print "Back in main\n"; sub test{ print "Entering test\n"; if (1) { my $foo = foo->new(); } if (1) { my $foo = foo->new(); } if (1) { my $foo = foo->new(); } print "Leaving test\n"; } __OUTPUT__ Entering test Leaving test DESTROYING 0 DESTROYING 1 DESTROYING 2 Back in Main
    UPDATE The point of showing this code, is that maybe it is not a memory leak, but merely a delay in clean up
Re: Strange memory growth?
by welchavw (Pilgrim) on Nov 07, 2003 at 21:33 UTC

      Ahh, now I see. There is the possibility that the if block is inside a loop, so perl will keep the lexical around so we don't have to recreate it on the next run through the loop.

      So if this is really a problem, you could always do:

      while($condition) { # do something $condition = 0; # Make sure condition is false at the end }

      Please, only if you've anaylized the program and you really are in danger of running out of memory (including swap space) from the conditionals you're doing. Anything else is a micro-optimization.

      ----
      I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
      -- Schemer

      : () { :|:& };:

      Note: All code is untested, unless otherwise stated

        Are you sure?... see my second post above. Even when I use the *same* *package* var, there is still unexplained memory growth. Your point about keeping the lexical around to avoid re-creating it next time through the loop doesn't seem applicable. In that case, I'm not using a lexical, nor am I creating a variable more than once.

        Update: Here's an even simpler example:

        $foo = 'perl' x 1000000; sleep 5; $foo = 'perl' x 1000000; sleep 5; $foo = 'perl' x 1000000; sleep 5; $foo = 'perl' x 1000000; sleep 5; $foo = 'perl' x 1000000; sleep 5; $foo = 'perl' x 1000000; sleep 5;

        Here I'm assigning the same value to the same variable over and over, why is memory use increasing? It seems too simple to be a bug, but still seems hard to explain.

        Please, only if you've anaylized the program and you really are in danger of running out of memory (including swap space) from the conditionals you're doing. Anything else is a micro-optimization.

        Possibly. But please remember when Perl is used in a mod_perl environment with a prefork MPM, you want to keep as much memory shared between children. This behaviour of Perl is very counter-productive in that environment: I'd rather take a little CPU hit for the overhead of creating the lexical again, and not unsharing any further memory, than have the current situation.

        More generally, I think the current behaviour of Perl to use more memory rather than more CPU, becomes more and more counter-productive as the speed of accessing memory is growing slower than the speed of the CPU with each new generation of systems.

        Liz

Re: Strange memory growth?
by tilly (Archbishop) on Nov 08, 2003 at 01:53 UTC
Re: Strange memory growth?
by ambrus (Abbot) on Nov 11, 2003 at 10:32 UTC

    Isn't that because ("perl" x 1000000) is a constant so it's subject to constant optimization, it's calculated at complile time?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://305427]
Approved by menolly
Front-paged by bart
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (6)
As of 2014-09-20 06:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (155 votes), past polls