Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

memory usage and leakage

by szabgab (Priest)
on Jun 23, 2011 at 10:43 UTC ( #911050=perlquestion: print w/ replies, xml ) Need Help??
szabgab has asked for the wisdom of the Perl Monks concerning the following question:

An application I am writing uses a lot of memory and there seem to be a serious memory leakage I try to track down. As I am far from being a memory expert I wonder if even I am looking at the right numbers and if the baseline information is correct? I am using the following script to check its own memory usage:
#!/usr/bin/perl use strict; use warnings; print_size('empty'); foreach my $module ( 'Data::Dumper', 'Scalar::Util', 'Getopt::Long', 'Net::XMPP', 'JSON::XS', 'LWP::UserAgent', 'POE qw(Component::Server::TCP)', 'Moose', ) { eval "use $module"; die $@ if $@; print_size("after $module"); } sub print_size { my ($msg) = @_; my @lines = qx{/bin/ps -e -o pid,ppid,vsize,rss,command | grep ^$$ +}; chomp @lines; foreach my $line (@lines) { my ($pid, $ppid, $vsize, $rss) = split /\s+/, $line; print "VM: $vsize RSS: $rss - $msg\n"; } return; }
and when running on SuSE with perl 5.10.0 I get the following data:
VM: 15972 RSS: 2324 - empty VM: 18740 RSS: 3256 - after Data::Dumper VM: 20968 RSS: 3388 - after Scalar::Util VM: 21760 RSS: 4240 - after Getopt::Long VM: 51348 RSS: 15408 - after Net::XMPP VM: 53648 RSS: 15588 - after JSON::XS VM: 57980 RSS: 17704 - after LWP::UserAgent VM: 65852 RSS: 21572 - after POE qw(Component::Server::TCP) VM: 88616 RSS: 31924 - after Moose
So if I understand correctly my script starts - after loading all the necessary modules - with 88 Mb memory used? Do I see correctly that loading Moose take 20Mb without even creating classes and objects? Net::XMPP is even bigger as it takes 30Mb ? Is the way I am measuring correct? As I'll need to run several instances of this process at the same time will they share some memory or will this number just be multiplied by the number of processes I have?

Comment on memory usage and leakage
Select or Download Code
Re: memory usage and leakage
by Sewi (Friar) on Jun 23, 2011 at 11:06 UTC

    The VM value is the address space allocated by an application, not the memory used. It doesn't give any reasonable result at all.

    RSS is the memory actually used and this is not shared across instances. The sum of RSS of all processes is the memory (including swap) used on your computer. Add kernel memory, disk cache and free and you got all of your memory.

    If you want to share some memory, "use" all required modules and start threading, but you need to check if all modules are thread-safe and ensure your code to be thread-safe.

    If you care about CPU or memory usage, remove PEO and Moose from your project as they eat up half of the scripts memory.

      Thanks, I especially liked your sig :)
      If you want to share some memory, "use" all required modules and start threading,

      Unless the application requires all the modules in all the threads, that is bad advice. It ensures that every thread will get it own copy of everything in every module, whether it ever uses it or not. And if the application does require all of those in every thread, it is badly designed and probably won't work anyway.

      If the OP structures his application properly, and requires only those modules he needs in each thread, it will reduce memory consumption conspicuously.

      But you're right that the simplest route to reducing memory consumption is to avoid behemoth frameworks that force you to load everything before you can do the simplest thing, and a shit-load more every time you want to do something extra.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        You're right as long as the threads do different things.

        But he said, he'd need many instances, my suggestion was to make every "instance" one "thread".

Re: memory usage and leakage
by Anonymous Monk on Jun 23, 2011 at 11:41 UTC
    Here is a win32 version, perl v5.12.2
    VM: 18660 < WS: 2756 VM: 608 > empty VM: 18660 < WS: 2956 VM: 636 > strict VM: 18660 < WS: 3092 VM: 760 > warnings VM: 18660 < WS: 3424 VM: 1024 > Module::Load VM: 18660 < WS: 3644 VM: 1216 > Module::Loaded VM: 20788 < WS: 4520 VM: 2036 > Module::Pluggable VM: 20972 < WS: 5084 VM: 2504 > Time::HiRes VM: 20972 < WS: 5168 VM: 2560 > Time::Local VM: 21020 < WS: 5556 VM: 2928 > Time::Piece VM: 21020 < WS: 5556 VM: 2928 > Time::Seconds VM: 21020 < WS: 5796 VM: 3172 > Time::gmtime VM: 21020 < WS: 5816 VM: 3184 > Time::localtime VM: 21020 < WS: 5816 VM: 3184 > Time::tm VM: 25152 < WS: 6544 VM: 3896 > Pod::Perldoc VM: 25152 < WS: 7660 VM: 4996 > Pod::Simple VM: 25220 < WS: 7996 VM: 5324 > Data::Dumper VM: 25220 < WS: 7996 VM: 5324 > Scalar::Util VM: 25220 < WS: 8404 VM: 5728 > Getopt::Long VM: 25352 < WS: 10188 VM: 7444 > LWP::UserAgent VM: 34232 < WS: 17548 VM: 14592 > Moose VM: 52272 < WS: 22800 VM: 19232 > POE => qw(Component::Server::TC +P)
memory leakage
by szabgab (Priest) on Jun 23, 2011 at 12:14 UTC
    I went further and tried the Net::XMPP code:
    #!/usr/bin/perl use strict; use warnings; print_size('empty'); use Net::XMPP; print_size("after Net::XMPP"); for (1..10) { my $conn = Net::XMPP::Client->new; print_size('Net::XMPP::Client created'); my $status = $conn->Connect( hostname => 'talk.google.com', port => 5222, componentname => 'gmail.com', connectiontype => 'tcpip', tls => 1, ); print_size('connection started'); $conn = undef; } sub print_size { my ($msg) = @_; my @lines = qx{/bin/ps -e -o pid,ppid,vsize,rss,command | grep ^$$ +}; chomp @lines; foreach my $line (@lines) { my ($pid, $ppid, $vsize, $rss) = split /\s+/, $line; print "VM: $vsize RSS: $rss - $msg\n"; } return; }
    The result is
    VM: 48596 RSS: 14648 - empty VM: 48596 RSS: 14652 - after Net::XMPP VM: 48808 RSS: 14744 - Net::XMPP::Client created VM: 67920 RSS: 18344 - connection started VM: 67920 RSS: 18344 - Net::XMPP::Client created VM: 68072 RSS: 18408 - connection started VM: 68072 RSS: 18408 - Net::XMPP::Client created VM: 68192 RSS: 18464 - connection started VM: 68192 RSS: 18464 - Net::XMPP::Client created VM: 68192 RSS: 18516 - connection started VM: 68192 RSS: 18516 - Net::XMPP::Client created VM: 68300 RSS: 18572 - connection started VM: 68300 RSS: 18572 - Net::XMPP::Client created VM: 68412 RSS: 18648 - connection started VM: 68412 RSS: 18648 - Net::XMPP::Client created VM: 68516 RSS: 18712 - connection started VM: 68516 RSS: 18712 - Net::XMPP::Client created VM: 68636 RSS: 18784 - connection started VM: 68636 RSS: 18784 - Net::XMPP::Client created VM: 68740 RSS: 18856 - connection started VM: 68740 RSS: 18856 - Net::XMPP::Client created VM: 68848 RSS: 18928 - connection started
    So if I understand correctly this means the Net::XMPP::Client is leaking memory? How could I better check it?
      Added
      use Devel::LeakGuard::Object qw(leakguard);
      and then leakguard around the for loop and I got:
      Object leaks found: Class Before After Delta FileHandle 0 10 10 IO::Select 0 20 20 IO::Socket::SSL 0 10 10 IO::Socket::SSL::SSL_Context 0 10 10 IO::Socket::SSL::SSL_HANDLE 0 10 10 Net::XMPP::Client 0 10 10 Net::XMPP::Debug 0 10 10 XML::Stream 0 10 10 XML::Stream::Parser 0 20 20 utf8 0 1 1
      I guess this means there is a memory leak.
Re: memory usage and leakage
by mirod (Canon) on Jun 23, 2011 at 12:46 UTC
    when running on SuSE with perl 5.10.0 I get the following data:

    Did you try using a different version of Perl? I don't follow p5p but I know by experience that 5.10.0 had some memory leaks that were fixed in 5.10.1 and above.

    perldelta for 5.10.1 in particular lists a weak reference to a hash would leak.

      Good idea. I have not tried yet as our production perl is 5.10.0 and I cannot change that any time soon. Besides it seems the leak is in one of the modules I am using.
        The leak in Perl can show up in one of the modules. It happened to XML::Twig, I had a couple of reports that it was leaking, and when I looked into it, it turned out to be a Perl problem. updating Perl to 5.10.1 fixed the problem for everyone.
Re: memory usage and leakage
by sundialsvc4 (Abbot) on Jun 23, 2011 at 13:36 UTC

    A few things to think about:

    1. Do you have a memory leak, or do you simply have memory consumption?   Test::Memory::Cycle is a good way to determine if you have self-referential data structures in a program.   (See also the other CPAN modules which it uses.)
    2. Is there memory pressure?   (If there isn’t, then the point-of-view of the always lazy operating system will be, “always room for one more!”   And, it is absolutely correct.)   The metrics produced by commands such as ps are only the roughest of indicators.   What you are really interested in is paging, particularly page steals, which is the actual indicator that something actually hurts.   If you don’t have pain, you don’t have a problem.
    3. Don’t guess ... measure.   If you suspect that excessive memory consumption is the root cause of your problem, demonstrate it.
    4. “Chips are cheap” now, whereas time is as expensive as ever.   I’m not talking about computer time here ... I’m talking about your own.   Sometimes the very best way to construct a program is to use the strongest software tools you can, then deal with any memory issues that actually develop (don’t borrow trouble ...) by “throwing silicon at it.”   After all, it’s only sand ...

      For 2 & 4:

      You're right - for web or short term applications. Net::XMPP is used for IM connections and those are usually long-life. If there is any real memory leak (which increases the used memory over time even without additional data being stored), it will become a problem, sooner or later.
      Let the final script run for a hour or day and everything might be fine, but if it's logged into XMPP for weeks or even longer, memory usage will grow endless.
      The problem is even worse, because there are running multiple instances (dosn't matter if they are processes or threads).

      Restarting the whole application from time to time is bad, even if the new process re-logins to XMPP after some seconds. The problem will catch you sooner or later even if you try to reboot the app depending on memory usage.

      1. Sure, there was a leak. I could see it even without Test::Memory::Cycle but using that module helped me locate it and fix it. Thanks for pointing in it out.

      2. Sure there is a pressure. We made some simple calculations of how much memory we will need and it it was more than what the servers can handle.

      3. Wasn't my question exactly about how to measure?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://911050]
Approved by moritz
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (8)
As of 2014-10-02 08:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    What is your favourite meta-syntactic variable name?














    Results (52 votes), past polls