Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

How to tell if script sucks...memory

by jriggs420 (Sexton)
on Feb 02, 2007 at 17:37 UTC ( [id://597971]=perlquestion: print w/replies, xml ) Need Help??

jriggs420 has asked for the wisdom of the Perl Monks concerning the following question:

I have heard that is possible for a perl script to *unintentionally* kill an OS, by using all the system's available memory or maxing out available processes. I believe failing to close file handles is one such example of how this can happen.
So my question is, how can you tell if your program is a memory hog, or is forking processes unbeknownst? Will 'ps -aux|grep me' show these processes. Will using the '-w' directive prevent it? I'm confident nothing I've written is a problem, some of the modules, not so much. Also, does it matter if the script runs as a cronjob?
I'm making the switch to mod_perl at some point so I rather find out sooner, rather than later these potential problems. I'm on Debian. Any insight into any of this would be greatly appreciated - J

Replies are listed 'Best First'.
Re: How to tell if script sucks...memory
by kyle (Abbot) on Feb 02, 2007 at 18:00 UTC

    how can you tell if your program is a memory hog, or is forking processes unbeknownst?

    You can fork processes using fork, system, or open. If you're running code you wrote, I'd expect you to know whether you'd forked anything off, so I'm guessing you're asking about detecting this in code you did not write.

    ps axl will show memory information (in VSZ and RSS fields). It also lists every process's parent process ID (in PPID), so for a given Perl program, you can see how much memory it's using, and you can look for other processes spawned from it.

    Will using the '-w' directive prevent it?

    No, that just warns about possible coding mistakes.

Re: How to tell if script sucks...memory
by zentara (Archbishop) on Feb 02, 2007 at 19:14 UTC
    You might be interested in my visual top in Tk at ztk-visual-top-w-kill or ztk-cpu-mem-logger for a single process. It will make it easy to spot cpu and mem spikes, and kill them quickly. Otherwise just watch the app with top itself. You could setup a monitoring script that will launch your app, then watch it's pid for excessive cpu or ram usage, maybe signalling you if it occurs. Another option is to set "nice" limits on the app, and if you notice it running at max nice level, then debug it.

    If you are only interested in monitoring a single app, you can try linux memory leak monitor. Then just put "use MeM;" in your script, and it will put a little Tk window in the screen corner, showing the current memory usage. I use it often to test my apps for memory leaks.


    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
Re: How to tell if script sucks...memory
by andyford (Curate) on Feb 02, 2007 at 18:07 UTC

    If you want to make sure that runaway programs don't take all available resources, most OS's provide means to limit resource usage on a per-process basis.

    non-Perl: Andy Ford

Re: How to tell if script sucks...memory
by Trizor (Pilgrim) on Feb 02, 2007 at 21:19 UTC
    If you want to check your memory use at several places in your program while it runs, this snippet provides a useful function to return how much memory you're using if you're on a *nix system.

    Sadly, this doesn't port to systems that lack a procfs.
    sub memsize { open(FH, '/proc/self/status'); while(<FH>) { return $_ if /^VmSize/; } close FH; }

      I've used this:

      sub my_mem { my @pslines = grep( m{ \A \s* (?: \d+ \s+ ){2} $$ \b }xms, `ps axl` ); die if ( scalar @pslines != 1 ); my $line = $pslines[0]; chomp $line; my @words = split /\s+/, $line; return $words[6]; }

      It relies on the system's ps axl, not procfs. You can also use Devel::Size to size up individual data structures at run time.

Re: How to tell if script sucks...memory
by AltBlue (Chaplain) on Feb 03, 2007 at 10:54 UTC
    As you mention Debian (a GNU/Linux distro), I'll throw in my .02$: check out the Linux namespace on CPAN. You'll find lots of Linux specific packages that could leverage your fiddlings. For instance, one module I personally tend to overuse from time to time is Linux::Smaps. Here's a sample usage:
    $ perl -MLinux::Smaps -le '$sm = Linux::Smaps->new($$); \ sub psize { $sm->update; print $sm->size; } \ psize(); require POSIX; psize(); POSIX->import; psize();' 6312 6560 7088
    If you want something even more fancy, try GTop. Here's a sample usage:
    $ perl -MGTop -le '$gtop = GTop->new; \ sub psize { print $gtop->proc_mem($$)->size; } \ psize(); require POSIX; psize(); POSIX->import; psize();' 7839744 7958528 8499200
Re: How to tell if script sucks...memory
by TOD (Friar) on Feb 03, 2007 at 05:13 UTC
    on a linux box there is a tiny tool names xosview.
Re: How to tell if script sucks...memory
by badaiaqrandista (Pilgrim) on Feb 04, 2007 at 14:23 UTC

    I have made a switch from cgi to mod_perl and memory is the least of my problem. My mod_perl application is now sized at 50M per process. That is 20 processes per server with 1G memory each.

    With the current normal memory size on servers ranging above 1Gb, you don't need to worry about your process sucking all the available memory. The only time you need to worry about running out of memory is when your application needs to process a really large data set, which needs to be in physical memory at the same time.

    If you really worried about your perl code unintentionally kill the OS by using up all the available memory, you might want to read about OOM killer on Linux.

    -cheepy-

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://597971]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chanting in the Monastery: (3)
As of 2024-07-13 09:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.