Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

taking memory snapshots with perl

by d_i_r_t_y (Monk)
on Mar 21, 2002 at 04:50 UTC ( #153227=perlquestion: print w/ replies, xml ) Need Help??
d_i_r_t_y has asked for the wisdom of the Perl Monks concerning the following question:

does anyone know how to get perl to take a reading of how much physical memory a process (namely, the current perl process) is using without using a (non-portable) system call? is this even possible in perl? specifically, i would like to use something along these lines to monitor for memory leaks in a mod_perl application.

any advice much appreciated.
matt

Comment on taking memory snapshots with perl
Re: taking memory snapshots with perl
by rob_au (Abbot) on Mar 21, 2002 at 05:36 UTC
    "without using a (non-portable) system call?"

    For *NIX-based systems, I strongly recommend the usage of Proc::ProcessTable which I have reviewed previously on this site here - This module however is not portable to Windows as it relies upon information derived from /proc/XXX/stat files (much nicer than parsing ps output). Without resorting to much XS magic, I believe that this would be your best bet - I would welcome correction on this with a link or example :-)

    Example usage of Proc::ProcessTable to determine a processes own memory utilisation, as either a definite size or a memory percentage, posted by myself can be at this node.

     

    perl -e 's&&rob@cowsnet.com.au&&&split/[@.]/&&s&.com.&_&&&print'

Re: taking memory snapshots with perl
by perrin (Chancellor) on Mar 21, 2002 at 06:11 UTC
    Well, you can look at how Apache::SizeLimit and Apache::GTopLimit do it, but this is not likely to help you find leaks. Perl allocates memory in chunks, and it grabs a bit ahead of what it's actually using. The instruction executed just before a large gain in memory is not necessarilly the one responsible for most of that memory getting used.

    The best way to find leaks is one that Matts described on the mod_perl list: run a section of code over and over, and watch the process with top to see if it grows over time. If you don't have any leaks, it will level off after maybe 100 or so requests and not grow anymore. If you do have leaks it will never stop growing.

    Matt tested his code by throwing an exception to jump out of his program right after the section of code he was testing, and he would keep moving the exception deeper into the code until he found the leaky part. Tedious, but effective.

Re: taking memory snapshots with perl
by PrakashK (Pilgrim) on Mar 21, 2002 at 19:04 UTC
    I haven't tested this under mod_perl, but I have used the following in FastCGI environment. It uses the Devel::Leak module.

    It does not give you the exact size of the memory allocated, rather gives you a number indicating a count of "things", where things mean scalar values, array and hash values etc. Check the man page for Devel::Leak for more info.

    package MemLeak; use strict; use vars qw(@ISA @EXPORT_OK); @ISA = qw(Exporter); @EXPORT_OK = qw(mlprint mlprintc); =head1 NAME MemLeak - Helper functions to track memory leaks =head1 SYNOPSIS use MemLeak qw(mlprint mlprintc); mlprint('BEGIN'); my $a = 5; mlprint('after $a'); my @a = (1..10); mlprintc('BEGIN', 'after @a'); my %a; @a{@a} = @a; mlprintc('BEGIN', 'after %a'); =cut use Devel::Leak; sub _time_now { my ($sec, $min, $hour, $day, $mon, $year) = localtime(time); return sprintf "%02d:%02d:%02d %02d/%02d/%02d", $hour, $min, $sec, $mon + 1, $day, $year - 100; } my $last_count = 0; my %CHECK_POINTS = ( _LAST => 0, ); sub _get_counts { my ($chkpt) = @_; my $handle; my $count = Devel::Leak::NoteSV($handle); my $delta = $count - $CHECK_POINTS{_LAST}; $CHECK_POINTS{_LAST} = $count; my $delta_cp; if ($chkpt) { if (exists $CHECK_POINTS{$chkpt}) { $delta_cp = $count - $CHECK_POINTS{$chkpt}; delete $CHECK_POINTS{$chkpt}; } else { $CHECK_POINTS{$chkpt} = $count; } } return ($count, $delta, $delta_cp); } =head1 DESCRIPTION =head2 mlprint(@args) print current SV count and change since last count =cut sub mlprint { my (@args) = @_; my ($count, $delta) = _get_counts; my $msg = join "", @args; chomp $msg if $msg; warn sprintf "[%d - %s] SV# = %d (%+d); %s\n", $$, _time_now, $count, $delta, $msg; } =pod =head2 mlprintc($chkpt, @args) print current SV count and changes since last count and since the checkpoint specified =cut sub mlprintc { my ($check_pt, @args) = @_; my ($count, $delta, $delta_cp) = _get_counts($check_pt); my $msg = join "", @args; chomp $msg if $msg; if (defined $delta_cp) { warn sprintf "[%d - %s] SV# = %d (%+d/%s: %+d); %s\n", $$, _time_now, $count, $delta, $check_pt, $delta_cp, $msg; } else { warn sprintf "[%d - %s] SV# = %d (%+d); %s: %s\n", $$, _time_now, $count, $delta, $check_pt, $msg; } } 1;
    Here's the output of the test snippet shown under SYNOPSIS above (on my debian GNU/Linux system):
    [32134 - 13:56:14 03/21/02] SV# = 4418 (+4418); BEGIN [32134 - 13:56:14 03/21/02] SV# = 4418 (+0); after $a [32134 - 13:56:14 03/21/02] SV# = 4428 (+10); BEGIN: after @a [32134 - 13:56:14 03/21/02] SV# = 4439 (+11/BEGIN: +11); after %a
    Add calls to mlprint and mlprintc at strategic places in your code to find out where memory is being allocated most.

    HTH,
    /prakash

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://153227]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (5)
As of 2014-12-28 07:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (179 votes), past polls