Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Memory Leak Package

by Bauldric (Novice)
on Oct 27, 2012 at 13:08 UTC ( #1001200=perlquestion: print w/ replies, xml ) Need Help??
Bauldric has asked for the wisdom of the Perl Monks concerning the following question:

Dear Reverends and God-Father of Perl

I am working with Perl 5.8.8 (ActiveState) under windows.

I have serious memory problems, although I allways use strict.pm (working only with local vars 'me $var') and do not use global vars. The program I've written has about 40000 Lines in more than 100 packages.

Do you know a good Package for searching memory leaks under Windows XP and perl version 5.8.8.

I would like to have something like the Devel::Symdump does to produce a list of all variables whose size has changed or are new allocated between two program-positions:

my $memdump = MemoryDump->new(); anyFunction(); print $memdump->diff(MemoryDump->new();

Exist something like that for ActiveState-Perl?

Thank you a lot for you response.

Update:

Thank you for your responses.

I've tried to install Devel::LeakTrace, but it does not work because I am working with a company-computer that prevents me to update any program (and I do not want bypass the restriction). So I've found that for counting References of objects the allready installed package 'B' works fine.

my $anon = []; my $otherref = $anon; my $refcount = B::svref_2object($anon)->REFCNT(); # => 2

Also the package 'Devel::Symdump' could I use, to trace the symbol-table. I've copied it into my private Class-Folder

I will inform you about as soon as I found the solution about my memory-leak. (And the way I've found it)

Comment on Memory Leak Package
Select or Download Code
Re: Memory Leak Package
by zwon (Monsignor) on Oct 27, 2012 at 13:59 UTC
Re: Memory Leak Package
by bulk88 (Priest) on Oct 27, 2012 at 15:43 UTC
    I think this thread is a continuation of How can I free the memory of a XML::Simple-Object.

    I've used Devel::Leak in the past on Win32 Perl.

    Why don't you try running your program on the latest ActivePerl (5.16) on a *different PC* and see if the memory usage is the same or not with your script? If it is much lower after the XML scalar obj is destroyed, you have your answer, upgrade.
Re: Memory Leak Package
by BrowserUk (Pope) on Oct 28, 2012 at 04:21 UTC

    Reading between the lines of your last couple of OPs, you seem to be parsing multiple XML files with XML::Simple; and doing so in threads in the hope of preventing leaks?

    From a couple of quick experiments, it seems that whichever underlying parser -- XML::SAX, XML::SAX::PurePerl, XML::Parser -- you use, they all leak substantial amounts of memory. XML::SAX more so that XML::Parser, but still in the range of 10 to 30MB per iteration for a 12MB xml file.

    My suggestion to cure that -- essentially what Zentara suggested earlier, but with a twist -- is to run the parser in a separate process. I used this to parse the file and then export the structure back to the parent:

    #! perl -slw use strict; use Storable qw[ freeze ]; use XML::Simple; $XML::Simple::PREFERRED_PARSER = 'XML::Parser'; binmode STDOUT; print freeze XMLin( $ARGV[ 0 ] );

    And then in the main script just a simple backticks command and thaw:

    for( 1 .. 1000 ) { my $xml = thaw `xmlSto.pl junk.xml`; print mem $_; <STDIN>; }

    It took about 1 extra second or so for the parent process to get access to the data structure, but it cures the leak completely.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

      Good morning, my holy reverend

      For my following little program I did no use threads

      use strict; use warnings; use CProcess; # own class use B; CProcess->PrintProcessInfo("perl.exe");

      # Output: Process-ID 12280: Memory 13716 KB (+13716 KB) - Peak 13716 KB - Threads 3 - Handles 115 (perl)

      The only packages I use are:

      use strict; use warnings; use Carp; use FileHandle; use Sys::Hostname; use Cwd; use Win32::Registry; use IO::Select; require Win32::Console; require Win32::Clipboard; require Win32::PerfLib; require Win32::OLE; require Tk::MainWindow; require 5.000; require Exporter;
      perhaps one of them uses threads.

      To count refcounts I use

      require B; my $anon = []; my $otherref = $anon; my $refcount = B::svref_2object($anon)->REFCNT(); # => 2

      which counts the referents of the referenced object (and not only of the variable itself like Devel::Peek)

        For my little program

        Is this the 400,000 line/100 package "small program"? :)

        I did not use threads

        I got the notion that you were using threads from your post:

        ID 7772: Memory 34200 KB ( 34200 KB) - Peak 34940 KB - Threads 3 - Han +dles 124 (perl) .......................................................^^^^^^^^^
        The only packages I use are: ... perhaps one of them uses threads.

        Hm. None of those use threads, (as in Perl Ithreads), that I am aware of.

        But Win32::OLE might well use -- or cause to be used -- kernel threads.

        But more interesting, is that none of those is XML::Simple as is the subject of How can I free the memory of a XML::Simple-Object?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

        Considered using VMMap on your perl process?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1001200]
Approved by moritz
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (3)
As of 2014-08-31 04:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls