Re: Memory Leak Package

[BrowserUk]

Reading between the lines of your last couple of OPs, you seem to be parsing multiple XML files with XML::Simple; and doing so in threads in the hope of preventing leaks?

From a couple of quick experiments, it seems that whichever underlying parser -- XML::SAX, XML::SAX::PurePerl, XML::Parser -- you use, they all leak substantial amounts of memory. XML::SAX more so that XML::Parser, but still in the range of 10 to 30MB per iteration for a 12MB xml file.

My suggestion to cure that -- essentially what Zentara suggested earlier, but with a twist -- is to run the parser in a separate process. I used this to parse the file and then export the structure back to the parent:

#! perl -slw
use strict;
use Storable qw[ freeze ];
use XML::Simple; $XML::Simple::PREFERRED_PARSER = 'XML::Parser'; 

binmode STDOUT;
print freeze XMLin( $ARGV[ 0 ] );

And then in the main script just a simple backticks command and thaw:

for( 1 .. 1000 ) {
    my $xml = thaw `xmlSto.pl junk.xml`;
    print mem $_; <STDIN>;
}

It took about 1 extra second or so for the parent process to get access to the data structure, but it cures the leak completely.

[Bauldric]

Good morning, my holy reverend

For my following little program I did no use threads

    use strict;
    use warnings;

    use CProcess; # own class
    use B;

    CProcess->PrintProcessInfo("perl.exe");

# Output: Process-ID 12280: Memory 13716 KB (+13716 KB) - Peak 13716 KB - Threads 3 - Handles 115 (perl)

The only packages I use are:

use strict;
use warnings;
use Carp;
use FileHandle;
use Sys::Hostname;
use Cwd;
use Win32::Registry;
use IO::Select;
require Win32::Console;
require Win32::Clipboard;
require Win32::PerfLib;
require Win32::OLE;
require Tk::MainWindow;
require 5.000;
require Exporter;

perhaps one of them uses threads.

To count refcounts I use

require B;

    my $anon = [];
    my $otherref = $anon;

    my $refcount =  B::svref_2object($anon)->REFCNT(); # => 2

which counts the referents of the referenced object (and not only of the variable itself like Devel::Peek)

[BrowserUk]

For my little program

Is this the 400,000 line/100 package "small program"? :)

I did not use threads

I got the notion that you were using threads from your post:

ID 7772: Memory 34200 KB ( 34200 KB) - Peak 34940 KB - Threads 3 - Han
+dles 124 (perl)
.......................................................^^^^^^^^^

The only packages I use are: ... perhaps one of them uses threads.

Hm. None of those use threads, (as in Perl Ithreads), that I am aware of.

But Win32::OLE might well use -- or cause to be used -- kernel threads.

But more interesting, is that none of those is XML::Simple as is the subject of How can I free the memory of a XML::Simple-Object?

[bulk88]

Considered using VMMap on your perl process?