Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: XML::Twig loves to eat my memory

by almut (Canon)
on Jul 22, 2010 at 20:02 UTC ( #850899=note: print w/ replies, xml ) Need Help??


in reply to XML::Twig loves to eat my memory

To narrow down on the issue, I would remove (comment out) everything not directly related to XML::Twig, and implement a dummy handler for BIOG.

In case the problem should disappear, step by step add back the original functionality until memory leaks again...


Comment on Re: XML::Twig loves to eat my memory
Re^2: XML::Twig loves to eat my memory
by carcus88 (Acolyte) on Jul 22, 2010 at 20:11 UTC
    I have tried this. Its not the database consuming the memory and when in debug I can watch the process size grow as I step through the twig parsing. Even if I break down the script into just XML::Twig parsing I can never get it to not consume memory even after $t->dispose() is called which according to the docs is supposed to work.

      In this case, create a minimal example that allows to replicate the problem (maybe with programmatically generated XML files), and submit a bug report — after having checked that you're using an up-to-date version of the module.

        Here is a much more minimal example. Both times I tried this once without the extra purge() and once with. Both times memory was at 166MB after the dispose().
        #!/usr/bin/perl -w use strict; use XML::Twig; my $inFile = '../data/2009/data_verified/abms_dialogb_10000000_1009999 +9.xml'; if ( ! $inFile ) { die("No input file specified"); } if ( ! -f $inFile ) { die("file '$inFile' not found"); } process($inFile); exit 0; # # Process the file # sub process { $inFile =~ /data_(\d+)_(\d+)/; my $t= new XML::Twig( TwigHandlers=> { BIOG => \&BIOG }, ); $t->parsefile( $inFile ); $t->purge(); # Will purge work? $t->dispose(); # Try to Free memory but does not work... } # # BIOG is XML element we are triggering # sub BIOG { my ($t, $BIOG)= @_; if ( ! checkBiog($BIOG->field('BIOG_NBR')) ) { print "Missing ". $BIOG->field('BIOG_NBR') . "\n"; } $t->purge(); # Tell XML::Twig to dispo of the rest of the tree we + don't care about return 1; } # # Check database for ID # sub checkBiog { my ($biog) = @_; return 1; }
        I will try to work on generating an example XML file. The current XML file is licensed data that I cannot share.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://850899]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (10)
As of 2014-04-18 00:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (460 votes), past polls