Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Memory usage of a "sub" in mod_perl

by diego_de_lima (Beadle)
on May 23, 2008 at 12:55 UTC ( #688132=perlquestion: print w/replies, xml ) Need Help??

diego_de_lima has asked for the wisdom of the Perl Monks concerning the following question:

Hello fellow monks!

Now that my mod_perl processes are gowing to more than 60mb each, Iīm trying to clean up averything I can. Well, it used to be 100mb each, and by eliminating some unused modules I could reduce ir drastically.

But now Iīm stucked in the problem of the process memory usage growing big during its lifetime. I have no public variables, and everything is OO. There is some packages which recieves big HTML strings which are latter printed to the browser. Sometimes I can have a 1mb HTML output or even more.

This packages are destroyed by the end of the session, wich is handled by a mod_perl handler (not a cgi file).

Iīm doing something like:

sub handler {
 my $r = shift;
 my $page = new Html::page;
 $page->{html} = '...and here it goes...';
 $page->end(); # Does some final processing
 print "Content-type:...".
       $page->{html};
 undef $page;
}
This "end" sub is very simple, and looks like:
sub end {
 my $class = shift;
 $class->{html} = "Header" . $class->{html} . "Footer";
}
Apache2::Satus says itīs consumes about 27kb before serving a request. But, after being executed, it grows to the size of the HTML it processed (109kb in the example below).

BEFORE serving a request I get:

Memory Usage for package Html::page
Totals: 68703 bytes, 67.09 Kb, 0.07 Mb | 1284 OPs
end                 27748 bytes | 573 OPs
new                  7045 bytes | 146 OPs
And AFTER serving a request I get:

Memory Usage for package Html::page
Totals: 151779 bytes, 148.22 Kb, 0.15 Mb | 1284 OPs
end                 109244 bytes | 573 OPs
new                   7045 bytes | 146 OPs
Obviously the mod_perl process grows in the same rate, and it never releases memory. So I end up sometimes with process that grows from 62mb to 120mb in just one request.

Am I doing something wrong? Acording to my knowledge, Perl should release the memory used in the $page->{html} variable just after I undef $page or simply do $page->{html} = ''; Or maybe Iīm wrong?

By the way: Linux Fedora 6 with Perl 5.8.8, Apache 2.2.3, mod_perl 2.0.2.

Thanks a lot for your attention monks!

Deigo de Lima

Replies are listed 'Best First'.
Re: Memory usage of a "sub" in mod_perl
by almut (Canon) on May 23, 2008 at 13:14 UTC

    You could try Devel::Cycle (or Test::Memory::Cycle, or Devel::Monitor) to check if your Html::page object maybe has circular references, which would stop the memory from being freed...  In that case, modules like Object::Destroyer might help to tackle the problem (unless you can get rid of the circular refs yourself otherwise).

    Update: Another thing to be aware of is that even without a true memory leak (due to circular refs), there is still the issue that memory once allocated by the Perl interpreter is usually not being returned to the OS before the interpreter process terminates — though this might be somewhat OS dependent (i.e. I've heard rumours that on Windows, memory actually may be returned to the OS — not sure though).

    The net effect is that - although memory is being freed and recycled Perl-internally - if you allocate a huge data structure (even if only once), the process size will stay at the maximum memory usage that ever occurred in the lifetime of the process...  I think there is a way around that problem by compiling Perl with the configuration settings to not use its own memory pool, but rather have it directly use the (slower) system malloc routines — In other words, if all else fails (and you're absolutely sure you don't have a classical memory leak), this might be worth a try...

Re: Momory usage of a "sub" in mod_perl
by moritz (Cardinal) on May 23, 2008 at 13:14 UTC
    Perhaps you have circular references somewhere that prevent structures from being garbage collected.

    Try Devel::Leek and Devel::Cycle to find them.

    Once you found them you can weak some of the references with Scalar::Util::weaken.

Re: Memory usage of a "sub" in mod_perl
by sundialsvc4 (Abbot) on May 23, 2008 at 13:32 UTC

    The operative question is:   does the memory consumption keep growing?

    If it does, then you probably have a circular-loop or something similar that is preventing garbage collection.

    But remember... a computer is lazy. (Lazier, in fact, than a good Perl programmer, if such a thing is possible...) It's not going to reduce the working-set size of a process unless there is actual competition for the storage-resource. There's no value, after all, in reclaiming a virtual-memory page that's going to incur a page-fault a few milliseconds later...

Re: Memory usage of a "sub" in mod_perl
by perrin (Chancellor) on May 23, 2008 at 17:07 UTC
    It is normal for a process to remain at the maximum size it has reached. Because you undef $page, that memory should be available to perl for other uses. If you see the process continue to grow, that's when you need to worry.

    I don't understand how 109K of HTML causes your process to grow 58MB. Do you actually have strings that take up that much space in a single request?

      Thnaks for all answers, they were quite usefull.

      Iīve many circular refs, and now I weakened them, which reduced slightly the memory usage. But still the process grows and donīt free its memory.

      100K HTML isnīt a big problem, but some times I have a 10mb ou 30mb structure in memory, which then generates a 3 mb HTML. And thats the problem.

      Iīm sure there arenīt circular refs anymore, but system doesnīt reclaims memory. I think this is the normal behavior, and Apache2::SizeLimit is the only solution for this. Am I right?

      But I still canīt undestand why it says the sub "end" is eating all this memory! I would understand if it says there is a global variable or something, but why a sub? For example, the perl-stats Memory Usage of the Html::page package says:
      128:       undef PV    44 bytes html
      129:       undef NULL  24 bytes 0xa7f4cb8
      130:       undef PV   37740 bytes Content-type: text/html
      Cache-Control: no-cache,no-store,max-age=0
      Pragma: no-cache
      
      <html>... ALL MY PAGE HERE 2 MB OF DATA!!!
      

      Why is it still arround? Iīve done a $page->{html} = ''; in the DESTROY method and Iīm sure it was invoked. And, by the way, it has nothing to do with the sub "end". "end" just parses $page->{html}, then flushes it to the browse and then undef $page->{html}. Shouldnīt "undef" be enouth?

      Thanks monks!

      Diego de Lima
        and then undef $page->{html}. Shouldn't "undef" be enough?

        That depends upon what that hash entry actually is. If it is a reference (or something containing a reference) to data which is held elsewhere, you just clear that slot from the reference, but not the referenced data.

        Consider:

        perl -le '$foo = "bar"; $h->{foo} = \$foo; undef $h->{foo}; print "foo + :$foo:"' foo :bar:

        which is different to

        perl -le '$foo = "bar"; $h->{foo} = \$foo; undef ${$h->{foo}}; print " +foo :$foo:"' foo ::

        where the referenced data is cleared.

        --shmem

        _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                      /\_¯/(q    /
        ----------------------------  \__(m.====·.(_("always off the crowd"))."·
        ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
        If you run the exact same request multiple times and the process size grows every time, you have a leak, probably a circular reference. If it stops growing but remains at the larger size, that is completely normal and the only way to avoid it is to structure your code so that it doesn't need to have all the data in memory at once.
Re: Memory usage of a "sub" in mod_perl
by Proclus (Beadle) on May 29, 2008 at 09:07 UTC
    Would you use a C based HTML template library like Clearsilver to play with your HTML files/templates? I'm currently working on a mod_perl/CGI::App/Clearsilver project, so far everything looks good. I don't have MB HTML files though.

    ------------------------------
    "geneva will not marry you no matter how much you love Larry Wall." Geneva Wall

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://688132]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2022-06-27 06:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My most frequent journeys are powered by:









    Results (86 votes). Check out past polls.

    Notices?