Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: out of memory issue while creating very large xml file using XML::Simple

by sundialsvc4 (Abbot)
on Jul 22, 2014 at 14:42 UTC ( #1094637=note: print w/ replies, xml ) Need Help??


in reply to out of memory issue while creating very large xml file using XML::Simple

I’ll presume that you are intending by the above to create a minimalistic example.   But, we need details here:   how large is this data structure?   Are you running a 64-bit version of Perl?   Do you have any idea as to the memory-footprint of this application prior to the step that generates the XML file?   Is there any possibility at all of self-referential “loops” within this data structure, which might cause an algorithm to iterate endlessly?   (Test::Memory::Cycle, and Devel::Cycle, can automatically look for these.)

Obviously, it is unreasonable to expect that a program running on a 32GB machine, dealing with a data structure that is thought to be about 1GB, and running 64-bit software in a 64-bit environment, would be encountering such issues.   (Whereas, BTW, if the actual system were 32-bit, the total memory space would be less than 2GB, and overflow is a distinct possibility.)   So, we really don’t have enough solid details here with which to offer many solutions . . . yet.   I’m not yet persuaded that we should finger this particular package.   Rather, I suspect that there is something else going on that we don’t yet see.


Comment on Re: out of memory issue while creating very large xml file using XML::Simple
Re^2: out of memory issue while creating very large xml file using XML::Simple
by perlCrazy (Monk) on Jul 22, 2014 at 16:48 UTC
    Thanks for response. When we increased the memory limit from 2gb to 20 GB for a process it went fine. datastructure was taking around 30 MB. we broke the datastructure and wrote to disk using storable module. main issue while writing xml file using XML::Simple. xmlOut method expects all contents in one data structure. it would have been better if we can write like streaming or inside loop. is there any way to create xml file in append mode using xml::simple?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1094637]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (9)
As of 2014-12-26 02:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (164 votes), past polls