Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

How can I free the memory of a XML::Simple-Object

by Bauldric (Novice)
on Oct 26, 2012 at 20:28 UTC ( #1001135=perlquestion: print w/ replies, xml ) Need Help??
Bauldric has asked for the wisdom of the Perl Monks concerning the following question:

Hi Perl-Monks

I am working with Perl 5.8.8 (ActiveState) under windows. I cannot change the perl-version. Furthermore I have to use the XML::Simple - Xml-Parser (which is the default parser)

No I have a memory problem. I cannot free the memory. using following code: (The Memory-Output is generated by a Class-Method of a self written class 'CProcess')

CProcess->PrintProcessInfo("perl.exe"); eval 'use XML::Simple'; CProcess->PrintProcessInfo("perl.exe"); my $xs = new XML::Simple(); my $root= $xs->XMLin('D:\xTools\test\xmltest.xml'); CProcess->PrintProcessInfo("perl.exe"); undef $root; undef $xs; eval 'no XML::Simple'; CProcess->PrintProcessInfo("perl.exe");
the output is

ID 7772: Memory 34200 KB ( 34200 KB) - Peak 34940 KB - Threads 3 - Handles 124 (perl)

ID 7772: Memory 35476 KB ( 1276 KB) - Peak 35476 KB - Threads 3 - Handles 124 (perl)

ID 7772: Memory 145492 KB (110016 KB) - Peak 145908 KB - Threads 3 - Handles 124 (perl)

ID 7772: Memory 133912 KB (-11580 KB) - Peak 145908 KB - Threads 3 - Handles 124 (perl)

Thank you a lot for helping me.

Comment on How can I free the memory of a XML::Simple-Object
Download Code
Re: How can I free the memory of a XML::Simple-Object
by Anonymous Monk on Oct 27, 2012 at 02:03 UTC

    I cannot free the memory. using following code:

    no, ie, unimport, doesn't free memory, that is not its purpose

    The Memory-Output is generated by a Class-Method of a self written class 'CProcess')

    What do the numbers mean?

Re: How can I free the memory of a XML::Simple-Object
by BrowserUk (Pope) on Oct 27, 2012 at 03:12 UTC

    The memory *is* freed. But it is freed back to the process memory pool; not back to the OS.

    This is normal. Asking the OS to allocate memory to a process is an expensive operation, so once the process has requested memory, it prefers to hang onto it when one part of the program is finished with it, so that it can be reused by a later part of the program.

    Think of it as memory caching. If every time you freed up a small chunk you gave it back to the OS; and then had to go back to the OS a split second later to request it back again for another part of the code; your program would run very slowly.

    There are exceptions. If your program requests a particularly large single chunk of memory, that may be allocated directly from the OS rather than the process memory pool; and given back as a single chunk when your program is done with it.

    But when you process an XML file into a nested data structure, the memory is not allocated all in one chunk, but rather in lots of small bits as the XML file is processed. This is done from the process memory pool, and when it is freed, it goes back to that pool.

    This is why you do not see any change in the size of the process' memory allocation, when you free the XML::Simple object. But rest assured, that memory is available to the rest of the program should it require it.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

      This is why you do not see any change in the size of the process' memory allocation, when you free te XML::Simple object. But reast assured, that memory is available to the rest of the program should it require it.
      The OP used (145908-35476)/4=27608 pages. Some of those pages HAVE to be returned to the OS and removed from the process, anything else is fragmentation leading to a DOS or a leak. All his memory could be speculative reserves for growth space for existing allocations by the memory allocator and therefore is not accessible.
        Some of those pages HAVE to be returned to the OS and removed from the process, anything else is fragmentation leading to a DOS or a leak. All his memory could be speculative reserves for growth space for existing allocations by the memory allocator and therefore is not accessible.

        First: I'll quote myself:

        There are exceptions....

        For example, If I allocate a 1e6 array:

        sub mem { my $mem = `tasklist /nh /fi "PID eq $$"`; $mem =~ tr[ \t\n][ ]s; return "$mem : $_[0]"; } print mem '1';; perl.exe 51704 Console 1 11,296 K : 1 $a[ $_ ] = $_ for 1 .. 1e6;; print mem '2';; perl.exe 51704 Console 1 43,684 K : 2 undef @a;; print mem '3';; perl.exe 51704 Console 1 35,468 K : 3
        • Allocating 1e6 array required 32MB.
        • After undefing it, 8MB is return to the OS, leaving 24MB in the memory pool.

        Why?

        Because the AV for 1e6 array of scalars requires a single chunk of contiguous memory to hold the 1e6 RV*'s. On my 64-bit system, pointers are 8-bytes; hence the AV requires a single allocation of 8*1e6 == 8MB. Because this is greater than some predefined size -- I think 1MB but that may vary between platforms or builds -- this part of the total memory allocation required by the array is treated specially and is "allocated directly from the OS rather than the process memory pool".

        However, the 1e6 SVs -- each 24 bytes -- are (individually) not greater than that limit, and are allocated from process memory pool.

        And there you have it: 8MB for the AV (in one chunk); 24MB for the scalars (in 1e6 x 24 byte chunks) giving 32MB total; of which 8MB is returned to the OS when freed and 24MB that isn't.

        Now to re-quote you:

        Some of those pages HAVE to be returned to the OS and removed from the process, anything else is fragmentation leading to a DOS or a leak. All his memory could be speculative reserves for growth space for existing allocations by the memory allocator and therefore is not accessible.

        Why "have to"? Where do you get that information from? What do you base that speculation on? What do those two sentences actually mean?

        Because I'm fairly knowledgeable about what Perl (on my platform) does with memory. Hard won knowledge; empirical evidence derived from practical, repeatable, demonstrable experiments; and I can not make any real sense of those two sentences, nor make them gel with what I know.

        If you have a better explanation; or can cite some reference; or provide some evidence in the form of a demonstration to support them; please share?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

      Thank you very much. Your statements helped me a lot. I see that at the second run, no new symbols will be added to the symbol-table. But why the memory is increased by 548 KB?
      use XML::Simple; use Devel::Symdump; Devel::Symdump->rnew(); # init CProcess->PrintProcessInfo("perl.exe"); { my $xs = new XML::Simple(); my $root= $xs->XMLin('D:\xTools\test\xmltest.xml'); } CProcess->PrintProcessInfo("perl.exe"); my $dump = Devel::Symdump->rnew(); { my $xs = new XML::Simple(); my $root= $xs->XMLin('D:\xTools\test\xmltest.xml'); } CProcess->PrintProcessInfo("perl.exe"); print "---------\nAdded new to symbol-table:\n"; print $dump->diff(Devel::Symdump->rnew()); print "---------\n";

      Process-ID 9372: Memory 34952 KB (+34952 KB) - Peak 34952 KB - Threads 3 - Handles 118 (perl)

      Process-ID 9372: Memory 133784 KB (+98832 KB) - Peak 145704 KB - Threads 3 - Handles 118 (perl)

      Process-ID 9372: Memory 134332 KB ( +548 KB) - Peak 146596 KB - Threads 3 - Handles 118 (perl)

      ---------

      Added new to symbol-table:

      ---------

        But why the memory is increased by 548 KB?

        Probably because you saved an object in $dump, where you didn't save one before (init)

        Probably for the same reason that it takes a little more memory to create an identical array the second time than it did the first:

        C:\test>p1 [0] Perl> sub mem { my $mem = `tasklist /nh /fi "PID eq $$"`; $mem =~ tr[ \t\n][ ]s; return "$mem : $_[0]"; };; print mem 1;; perl.exe 291940 Console 1 11,292 K : 1 $a[ $_ ] = $_ for 1 .. 1e6;; print mem 2;; perl.exe 291940 Console 1 43,680 K : 2 undef @a;; print mem 3;; perl.exe 291940 Console 1 35,464 K : 3 $a[ $_ ] = $_ for 1 .. 1e6;; print mem 4;; perl.exe 291940 Console 1 44,848 K : 4

        When building an array piecemeal this way, the base AV starts with 8 elements, therefore requires 64 bytes contiguous memory be allocated. When you add the 9th element to it, the AV has to be reallocate to accommodate that new element and so a new chunk of memory double the size (128 bytes) is allocated; the existing 8 elements are copied across and the new 9th element is added (leaving space for 7 more new elements). Then and the first AV is freed back to the memory pool.

        Then when you go to add the 17th element; the size is doubled again (256 byte AV is allocated) and the 128 are freed back to the pool. This process of doubling continues, until the size of the size of the AV required gets bigger than the pool limit, and then rather than allocating the newly doubled AV from the pool; it is allocated from the OS.

        During the process of creating the array the first time, the pool is expanded -- by requesting more memory from the OS -- to accommodate the AVs when they are less than the size where they get allocated directly from the OS. And when they are done with, those allocations are returned to the pool and will be reused for other allocations including the many 24-bytes SVs. by the time the first array is freed, the once contiguous chunks have been re-allocated to smaller subdivisions.

        Hence, when the array is built the second time, the continuous chunks originally allocated from the OS to accommodate the intermediate AVs are no longer available as contiguous chunks, so as the array is built the second time, it is necessary to go back to the OS for new contiguous allocations, despite that the final array will be the same size as the first.

        Hashes go through similar processes as they expand.

        As XML::Simple builds nested structures of hashes and arrays, it has similar requirements for contiguous chunks of memory, that may need to be reallocated when re-building the same data-structure a second time.

        (Note for the pedants: The above may not be a totally accurate description of the process; but it is sufficient to explain the point I am trying to convey.)


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

Re: How can I free the memory of a XML::Simple-Object
by Anonymous Monk on Oct 27, 2012 at 03:16 UTC
    Try
    CProcess->PrintProcessInfo("perl.exe"); eval 'use XML::Simple'; CProcess->PrintProcessInfo("perl.exe"); eval 'no XML::Simple'; CProcess->PrintProcessInfo("perl.exe"); my $xs = new XML::Simple(); CProcess->PrintProcessInfo("perl.exe"); my $root= $xs->XMLin('D:\xTools\test\xmltest.xml'); CProcess->PrintProcessInfo("perl.exe"); undef $xs; CProcess->PrintProcessInfo("perl.exe"); undef $root; CProcess->PrintProcessInfo("perl.exe"); print "\n##########\n"; my $xs = new XML::Simple(); CProcess->PrintProcessInfo("perl.exe"); my $root= $xs->XMLin('D:\xTools\test\xmltest.xml'); CProcess->PrintProcessInfo("perl.exe"); undef $xs; CProcess->PrintProcessInfo("perl.exe"); undef $root; CProcess->PrintProcessInfo("perl.exe");
Re: How can I free the memory of a XML::Simple-Object
by bulk88 (Priest) on Oct 27, 2012 at 03:17 UTC
    " eval 'no XML::Simple';" that probably did nothing. XML::Simple probably does not have or inherit an unimport sub. You can always ->import() ->unimport() any package and it will always "work", because of a special exception in the interp for those 2 method calls.
    CProcess->PrintProcessInfo("perl.exe"); eval 'use XML::Simple'; CProcess->PrintProcessInfo("perl.exe"); { my $xs = new XML::Simple(); my $root= $xs->XMLin('D:\xTools\test\xmltest.xml'); CProcess->PrintProcessInfo("perl.exe"); undef $root; undef $xs; } CProcess->PrintProcessInfo("perl.exe");
    Try that. Perl will always destroy at a scope boundary/change. Reassigning may or may not trigger a DESTROY (I don't exactly).

    If you are desperate on memory you can try Symbol's delete_package and/or unloading XS modules (I'm not describing how to do that at the moment). Neither (delete_package/unloading an XS module) are guaranteed to free all memory. With a Perl that old, there are probably leaks that you can't fix without a newer version of Perl.

    I also see "Threads 3", are you using ithreads? On a Perl that old they are alpha.
Re: How can I free the memory of a XML::Simple-Object
by zentara (Archbishop) on Oct 27, 2012 at 09:17 UTC
    how can I free the memory?

    Figure out how to do this with forks instead of threads. Forking returns memory always.


    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
      He is running on Windows. Each psuedofork will allocate memory.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1001135]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (8)
As of 2014-07-12 11:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (239 votes), past polls