Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

module memory usage

by smackdab (Pilgrim)
on Dec 22, 2003 at 00:30 UTC ( #316278=perlquestion: print w/ replies, xml ) Need Help??
smackdab has asked for the wisdom of the Perl Monks concerning the following question:

Is there a module or snippet that I can look at to show me approx. how much a module uses in memory? I was thinking about looping through the ones I have included, but I don't know of fn to call to do it...

Kinda what I'm looking to do:
foreach (@mods) { eval { $premem = ?; require "$mod.pm"; import "$mod.pm"; $postmem = ?; Symbol::delete_package($mod); #saw in faq ;) print "Mod: $mod, used: ", $postmem-$premem, "\n"; } }

Comment on module memory usage
Download Code
Re: module memory usage
by edoc (Chaplain) on Dec 22, 2003 at 00:45 UTC
Re: module memory usage
by Joost (Canon) on Dec 22, 2003 at 00:54 UTC
    First, a couple of remarks:
    • How much memory a modules uses is generally dependend on what you do with it, after you load it.
    • Deleting packages is unreliable AFAIK.
    • Freeing variables, namespaces etc. in Perl does NOT in general decrease the amount of memory taken by the perl process.

    I do not know of a standard way of accessing this data, but when I have a need for it, I usually run top (or equivalent resource monitor for other systems), and execute the code to see how much it takes up. You can make it a little easier by sending out some messages about what the program is doing (i.e. which modules are being loaded) and then sleep()ing for a couple of seconds. I haven't found a real need for anything more sophisticated.

    YMMV
    Joost.

Re: module memory usage
by stvn (Monsignor) on Dec 22, 2003 at 01:51 UTC

    This is from the ModPerl Cookbook, pulled from pages 317-322 (i'm quoting mixed with some paraphrasing).

    The B::Size and B::TerseSize modules, both part of the B::Size distribution, include a number of routines to calculate memory usage of perl packages, subroutines and variables.

    It goes on to describe how to set it up with Apache::Status on a mod_perl server to get useful information displayed by Apache::Status. If you would like more info on that, msg me and I will give it to you. But if you are not running mod_perl, then i would just look at the B::Size documentation. I cannot speak for it, as i have never used or known of anyone who has used it, but this Cookbook has never let me down before, so I weigh its suggestions pretty highly.

    The book then goes on to talk about reducing memory consumption by not exporting symbols from modules, basically doing this:

    use POSIX (); # instead of this ... use POSIX;

    POSIX imports over 560 symbols, each imported symbol being about a 120 byte memory penalty. According to this book that works out to 140KB of memery overhead that could be avoided with 2 colons and a little more typing. Not alot in a small one-off script, but that piles up fast when you have 50-100 mod_perl/Apache child processes screaming for attention from the CPU.

    But as anyone who has been around in this biz for over 18 months (the cycle time for Moore's Law) will tell you. Don't worry about stuff like this too much unless you know for absolutely sure its a problem, because chances are in another 18 months,... well you know.

    -stvn
      POSIX imports over 560 symbols, each imported symbol being about a 120 byte memory penalty. According to this book that works out to 140KB of memery overhead that could be avoided with 2 colons and a little more typing. Not alot in a small one-off script, but that piles up fast when you have 50-100 mod_perl/Apache child processes screaming for attention from the CPU.

      You'll also want to realize that the qualifiied name adds a handful of additional dereferences and a hash lookup when you could have spent the memory on a local access.

      --
        Devil
        Devil,

        Quite true, the classic CPU versus memory tradeoff. As with all "optimizations", you almost never get away for free.

        Personally I would rather waste a CPU cycle or 2 since they tend to be very cheap, and for what I do (mod_perl web apps) a pico second here or there is usually okay, while running into disk swap for memory is not.

        In the end, I stick with the old mantra:

        Premature optimization is the route of all evil.

        - C.A. Hoare (although usually attributed to Donald Knuth)

        -stvn
        What kind of genius has 50-100 mod_perl/Apache child processes?
Re: module memory usage
by exussum0 (Vicar) on Dec 22, 2003 at 02:36 UTC
    You can't free() memory in perl. You can deallocate it from what perl is actively using into a passive mode, where it will be reallocated for more in-use stuff later.

    In short, memory for a running perl program either remains at a size or grows depending if more memory is needed and there is no free memory laying aboot.


    Play that funky music white boy..
Re: module memory usage
by gmpassos (Priest) on Dec 22, 2003 at 07:14 UTC
    I have taked a look in the module B::Size, as our friend stvn pointed it, and based in a code of Apache::Status I made this function that prints a package size report:
    package_size('main'); sub package_size { my($package) = shift ; require B::TerseSize ; print("Memory Usage for package $package\n\n"); my($subs, $opcount, $opsize) = B::TerseSize::package_size($package +); print("Totals: $opsize bytes | $opcount OPs\n\n"); my($clen, $slen, $nlen); my @keys = map { $nlen = length > $nlen ? length : $nlen; $_; } ( sort { $subs->{$b}->{size} <=> $subs->{$a}->{size +} } keys %$subs ); $clen = length $subs->{$keys[0]}->{count}; $slen = length $subs->{$keys[0]}->{size}; for my $name (@keys) { my $stats = $subs->{$name}; if ($name =~ /^my /) { printf "%-${nlen}s %${slen}d bytes\n", $name, $stats->{size} ; } else { printf "%-${nlen}s %${slen}d bytes | %${clen}d OPs\n", $name, +$stats->{size}, $stats->{count} ; } } }

    And here's an output example:

    Memory Usage for package main Totals: 24699 bytes | 212 OPs package_size 9526 bytes | 212 OPs *ENV{HASH} 3372 bytes | 0 OPs *SIG{HASH} 2943 bytes | 0 OPs *INC{HASH} 1378 bytes | 0 OPs *_{HASH} 228 bytes | 0 OPs *INC{ARRAY} 175 bytes | 0 OPs *_{ARRAY} 125 bytes | 0 OPs *0{SCALAR} 69 bytes | 0 OPs *1{SCALAR} 69 bytes | 0 OPs *ARGV{ARRAY} 60 bytes | 0 OPs *_{SCALAR} 26 bytes | 0 OPs

    Enjoy!

    Graciliano M. P.
    "Creativity is the expression of the liberty".

      Very cool, I think it shows the size of the code loaded...I wonder if there is a way to get the runtime footprint?

      While Tk is worth every byte, It uses:
      Memory: main Totals: 52513 bytes | 225 OPs Memory: Tk Totals: 154632 bytes | 1663 OPs Memory: Tk::Adjuster Totals: 99843 bytes | 2000 OPs Memory: Tk::Balloon Totals: 212545 bytes | 2939 OPs Memory: Tk::BrowseEntry Totals: 126060 bytes | 2519 OPs Memory: Tk::Canvas Totals: 22194 bytes | 244 OPs Memory: Tk::CursorControl Totals: 161874 bytes | 2843 OPs Memory: Tk::Entry Totals: 122144 bytes | 2324 OPs Memory: Tk::ItemStyle Totals: 6995 bytes | 91 OPs Memory: Tk::JPEG Totals: 1911 bytes | 1 OPs Memory: Tk::Label Totals: 1824 bytes | 11 OPs Memory: Tk::LabEntry Totals: 4822 bytes | 77 OPs Memory: Tk::MListbox Totals: 214159 bytes | 3940 OPs Memory: Tk::Menu Totals: 193278 bytes | 4004 OPs Memory: Tk::Pixmap Totals: 2601 bytes | 6 OPs Memory: Tk::ROText Totals: 8104 bytes | 127 OPs Memory: Tk::Scrollbar Totals: 31046 bytes | 472 OPs Grand Total: 1416545
      And running the above and using taskman (win32) to view the footprint, it give ~8megs (base perl takes an extra 1.7m)...(unless my poor math skills have affected my thinking ;-)
Re: module memory usage
by Juerd (Abbot) on Dec 22, 2003 at 18:15 UTC

    Here's a nasty trick that I use:

    perl -le'require The::Module; print +(split " ", `cat /proc/$$/stat`)[ +22]'
    substract the value you get without the require statement, et voila. As far as I know, this works only under Linux. I've used it for the documentation of Exporter::Tidy.

    Juerd # { site => 'juerd.nl', plp_site => 'plp.juerd.nl', do_not_use => 'spamtrap' }

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://316278]
Approved by rob_au
Front-paged by broquaint
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (6)
As of 2014-11-27 18:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (186 votes), past polls