module memory usage

[smackdab]

Is there a module or snippet that I can look at to show me approx. how much a module uses in memory? I was thinking about looping through the ones I have included, but I don't know of fn to call to do it...

Kinda what I'm looking to do:

foreach (@mods) {
  eval { 
    $premem = ?;
    require "$mod.pm"; import "$mod.pm"; 
    $postmem = ?;
    Symbol::delete_package($mod); #saw in faq ;)
    print "Mod: $mod, used: ", $postmem-$premem, "\n";
  }
}

[stvn]

This is from the ModPerl Cookbook, pulled from pages 317-322 (i'm quoting mixed with some paraphrasing).

The B::Size and B::TerseSize modules, both part of the B::Size distribution, include a number of routines to calculate memory usage of perl packages, subroutines and variables.

It goes on to describe how to set it up with Apache::Status on a mod_perl server to get useful information displayed by Apache::Status. If you would like more info on that, msg me and I will give it to you. But if you are not running mod_perl, then i would just look at the B::Size documentation. I cannot speak for it, as i have never used or known of anyone who has used it, but this Cookbook has never let me down before, so I weigh its suggestions pretty highly.

The book then goes on to talk about reducing memory consumption by not exporting symbols from modules, basically doing this:

use POSIX ();

# instead of this ...

use POSIX;

POSIX imports over 560 symbols, each imported symbol being about a 120 byte memory penalty. According to this book that works out to 140KB of memery overhead that could be avoided with 2 colons and a little more typing. Not alot in a small one-off script, but that piles up fast when you have 50-100 mod_perl/Apache child processes screaming for attention from the CPU.

But as anyone who has been around in this biz for over 18 months (the cycle time for Moore's Law) will tell you. Don't worry about stuff like this too much unless you know for absolutely sure its a problem, because chances are in another 18 months,... well you know.

-stvn

[diotalevi]

POSIX imports over 560 symbols, each imported symbol being about a 120 byte memory penalty. According to this book that works out to 140KB of memery overhead that could be avoided with 2 colons and a little more typing. Not alot in a small one-off script, but that piles up fast when you have 50-100 mod_perl/Apache child processes screaming for attention from the CPU.

You'll also want to realize that the qualifiied name adds a handful of additional dereferences and a hash lookup when you could have spent the memory on a local access.

--
  Devil

[stvn]

Devil,

Quite true, the classic CPU versus memory tradeoff. As with all "optimizations", you almost never get away for free.

Personally I would rather waste a CPU cycle or 2 since they tend to be very cheap, and for what I do (mod_perl web apps) a pico second here or there is usually okay, while running into disk swap for memory is not.

In the end, I stick with the old mantra:

Premature optimization is the route of all evil.

- C.A. Hoare (although usually attributed to Donald Knuth)

-stvn

[Anonymous Monk]

What kind of genius has 50-100 mod_perl/Apache child processes?

[edoc]

maybe checkout ModPerl Guide - Measuring the Memory of the Process

[Joost]

First, a couple of remarks:
• How much memory a modules uses is generally dependend on what you do with it, after you load it.
• Deleting packages is unreliable AFAIK.
• Freeing variables, namespaces etc. in Perl does NOT in general decrease the amount of memory taken by the perl process.

I do not know of a standard way of accessing this data, but when I have a need for it, I usually run top (or equivalent resource monitor for other systems), and execute the code to see how much it takes up. You can make it a little easier by sending out some messages about what the program is doing (i.e. which modules are being loaded) and then sleep()ing for a couple of seconds. I haven't found a real need for anything more sophisticated.

YMMV
Joost.

[gmpassos]

I have taked a look in the module B::Size, as our friend stvn pointed it, and based in a code of Apache::Status I made this function that prints a package size report:

  package_size('main');
    
  sub package_size {
    my($package) = shift ;
    
    require B::TerseSize ;
  
    print("Memory Usage for package $package\n\n");
  
    my($subs, $opcount, $opsize) = B::TerseSize::package_size($package
+);
  
    print("Totals: $opsize bytes | $opcount OPs\n\n");
  
    my($clen, $slen, $nlen);
  
    my @keys = map { $nlen = length > $nlen ? length : $nlen; $_; }
                   ( sort { $subs->{$b}->{size} <=> $subs->{$a}->{size
+} } keys %$subs );
  
    $clen = length $subs->{$keys[0]}->{count};
    $slen = length $subs->{$keys[0]}->{size};
  
    for my $name (@keys) {
      my $stats = $subs->{$name};
      if ($name =~ /^my /) {
        printf "%-${nlen}s %${slen}d bytes\n", $name, $stats->{size} ;
      }
      else {
        printf "%-${nlen}s %${slen}d bytes | %${clen}d OPs\n", $name, 
+$stats->{size}, $stats->{count} ;
      }
    }
  }

And here's an output example:

Memory Usage for package main

Totals: 24699 bytes | 212 OPs

package_size 9526 bytes | 212 OPs
*ENV{HASH}   3372 bytes |   0 OPs
*SIG{HASH}   2943 bytes |   0 OPs
*INC{HASH}   1378 bytes |   0 OPs
*_{HASH}      228 bytes |   0 OPs
*INC{ARRAY}   175 bytes |   0 OPs
*_{ARRAY}     125 bytes |   0 OPs
*0{SCALAR}     69 bytes |   0 OPs
*1{SCALAR}     69 bytes |   0 OPs
*ARGV{ARRAY}   60 bytes |   0 OPs
*_{SCALAR}     26 bytes |   0 OPs

Enjoy!

[smackdab]

Very cool, I think it shows the size of the code loaded...I wonder if there is a way to get the runtime footprint?

While Tk is worth every byte, It uses:

Memory: main    Totals: 52513 bytes | 225 OPs
Memory: Tk      Totals: 154632 bytes | 1663 OPs
Memory: Tk::Adjuster    Totals: 99843 bytes | 2000 OPs
Memory: Tk::Balloon     Totals: 212545 bytes | 2939 OPs
Memory: Tk::BrowseEntry Totals: 126060 bytes | 2519 OPs
Memory: Tk::Canvas      Totals: 22194 bytes | 244 OPs
Memory: Tk::CursorControl       Totals: 161874 bytes | 2843 OPs
Memory: Tk::Entry       Totals: 122144 bytes | 2324 OPs
Memory: Tk::ItemStyle   Totals: 6995 bytes | 91 OPs
Memory: Tk::JPEG        Totals: 1911 bytes | 1 OPs
Memory: Tk::Label       Totals: 1824 bytes | 11 OPs
Memory: Tk::LabEntry    Totals: 4822 bytes | 77 OPs
Memory: Tk::MListbox    Totals: 214159 bytes | 3940 OPs
Memory: Tk::Menu        Totals: 193278 bytes | 4004 OPs
Memory: Tk::Pixmap      Totals: 2601 bytes | 6 OPs
Memory: Tk::ROText      Totals: 8104 bytes | 127 OPs
Memory: Tk::Scrollbar   Totals: 31046 bytes | 472 OPs
Grand Total: 1416545

And running the above and using taskman (win32) to view the footprint, it give ~8megs (base perl takes an extra 1.7m)...(unless my poor math skills have affected my thinking ;-)

[exussum0]

You can't free() memory in perl. You can deallocate it from what perl is actively using into a passive mode, where it will be reallocated for more in-use stuff later.

In short, memory for a running perl program either remains at a size or grows depending if more memory is needed and there is no free memory laying aboot.

[Juerd]

Here's a nasty trick that I use:

perl -le'require The::Module; print +(split " ", `cat /proc/$$/stat`)[
+22]'

substract the value you get without the require statement, et voila. As far as I know, this works only under Linux. I've used it for the documentation of Exporter::Tidy.