Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: XML::Twig loves to eat my memory

by almut (Canon)
on Jul 22, 2010 at 20:02 UTC ( #850899=note: print w/ replies, xml ) Need Help??


in reply to XML::Twig loves to eat my memory

To narrow down on the issue, I would remove (comment out) everything not directly related to XML::Twig, and implement a dummy handler for BIOG.

In case the problem should disappear, step by step add back the original functionality until memory leaks again...


Comment on Re: XML::Twig loves to eat my memory
Re^2: XML::Twig loves to eat my memory
by carcus88 (Acolyte) on Jul 22, 2010 at 20:11 UTC
    I have tried this. Its not the database consuming the memory and when in debug I can watch the process size grow as I step through the twig parsing. Even if I break down the script into just XML::Twig parsing I can never get it to not consume memory even after $t->dispose() is called which according to the docs is supposed to work.

      In this case, create a minimal example that allows to replicate the problem (maybe with programmatically generated XML files), and submit a bug report — after having checked that you're using an up-to-date version of the module.

        Here is a much more minimal example. Both times I tried this once without the extra purge() and once with. Both times memory was at 166MB after the dispose().
        #!/usr/bin/perl -w use strict; use XML::Twig; my $inFile = '../data/2009/data_verified/abms_dialogb_10000000_1009999 +9.xml'; if ( ! $inFile ) { die("No input file specified"); } if ( ! -f $inFile ) { die("file '$inFile' not found"); } process($inFile); exit 0; # # Process the file # sub process { $inFile =~ /data_(\d+)_(\d+)/; my $t= new XML::Twig( TwigHandlers=> { BIOG => \&BIOG }, ); $t->parsefile( $inFile ); $t->purge(); # Will purge work? $t->dispose(); # Try to Free memory but does not work... } # # BIOG is XML element we are triggering # sub BIOG { my ($t, $BIOG)= @_; if ( ! checkBiog($BIOG->field('BIOG_NBR')) ) { print "Missing ". $BIOG->field('BIOG_NBR') . "\n"; } $t->purge(); # Tell XML::Twig to dispo of the rest of the tree we + don't care about return 1; } # # Check database for ID # sub checkBiog { my ($biog) = @_; return 1; }
        I will try to work on generating an example XML file. The current XML file is licensed data that I cannot share.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://850899]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (12)
As of 2015-07-01 21:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (22 votes), past polls