http://www.perlmonks.org?node_id=1001194


in reply to Re^2: How can I free the memory of a XML::Simple-Object
in thread How can I free the memory of a XML::Simple-Object

Probably for the same reason that it takes a little more memory to create an identical array the second time than it did the first:

C:\test>p1 [0] Perl> sub mem { my $mem = `tasklist /nh /fi "PID eq $$"`; $mem =~ tr[ \t\n][ ]s; return "$mem : $_[0]"; };; print mem 1;; perl.exe 291940 Console 1 11,292 K : 1 $a[ $_ ] = $_ for 1 .. 1e6;; print mem 2;; perl.exe 291940 Console 1 43,680 K : 2 undef @a;; print mem 3;; perl.exe 291940 Console 1 35,464 K : 3 $a[ $_ ] = $_ for 1 .. 1e6;; print mem 4;; perl.exe 291940 Console 1 44,848 K : 4

When building an array piecemeal this way, the base AV starts with 8 elements, therefore requires 64 bytes contiguous memory be allocated. When you add the 9th element to it, the AV has to be reallocate to accommodate that new element and so a new chunk of memory double the size (128 bytes) is allocated; the existing 8 elements are copied across and the new 9th element is added (leaving space for 7 more new elements). Then and the first AV is freed back to the memory pool.

Then when you go to add the 17th element; the size is doubled again (256 byte AV is allocated) and the 128 are freed back to the pool. This process of doubling continues, until the size of the size of the AV required gets bigger than the pool limit, and then rather than allocating the newly doubled AV from the pool; it is allocated from the OS.

During the process of creating the array the first time, the pool is expanded -- by requesting more memory from the OS -- to accommodate the AVs when they are less than the size where they get allocated directly from the OS. And when they are done with, those allocations are returned to the pool and will be reused for other allocations including the many 24-bytes SVs. by the time the first array is freed, the once contiguous chunks have been re-allocated to smaller subdivisions.

Hence, when the array is built the second time, the continuous chunks originally allocated from the OS to accommodate the intermediate AVs are no longer available as contiguous chunks, so as the array is built the second time, it is necessary to go back to the OS for new contiguous allocations, despite that the final array will be the same size as the first.

Hashes go through similar processes as they expand.

As XML::Simple builds nested structures of hashes and arrays, it has similar requirements for contiguous chunks of memory, that may need to be reallocated when re-building the same data-structure a second time.

(Note for the pedants: The above may not be a totally accurate description of the process; but it is sufficient to explain the point I am trying to convey.)


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

Replies are listed 'Best First'.
Re^4: How can I free the memory of a XML::Simple-Object
by Bauldric (Novice) on Oct 27, 2012 at 12:34 UTC
    You are really great :-) By the way. Do you know a good Package for searching memory leaks under Windows XP and perl version 5.8.8. I have memory problems, although I use allways strict.pm (working only with local vars 'me $var') and have no global vars. The program has about 40000 Lines in 100 packages. I would like to have something like the symdump does to produce a list of all variables whose size has changed or are new allocated between two program-positions:
    my $memdump = MemoryDump->new(); anyFunction(); print $memdump->diff(MemoryDump->new();

    Exist something like that for ActiveState-Perl?

    For your example which you've wrote for me: So if I set the size of the array @a

    my @a = (); $#a = 1e6;

    the memory will be used again

    my @a = (); $#a = 1e6; mem(1); $a[ $_ ] = $_ for 1 .. 1e6; mem(2); undef @a; mem(3); my @a = (); $#a = 1e6; $a[ $_ ] = $_ for 1 .. 1e6; mem(4); sub mem { my $mem = `tasklist /nh /fi "PID eq $$"`; $mem =~ tr[ \t\n][ ]s; print "$mem : $_[0]\n"; };
      Exist something like that for ActiveState-Perl?

      See Devel::Size & Devel::Size::Report which does/can do something similar.

      (I also just noticed Devel::SizeMe which I've no knowledge of, but comes from a good author, so would be worth investigating.)

      I have to admit that I rarely resort to such tools for tracking memory leaks. Especially if threads are involved; I have my own method.

      Basically this involves commenting out the bodies of suspect subroutines/methods -- but having them return plausibly 'good' values -- one at a time until I find which one is contributing to the leak. Then zeroing in from there.

      With a program as complex as yours, it could be quite a laborious process, but I find it quicker than wading through reams of dumps. Especially where threads are involved which usually forces you to have to cross-relate data from several threads.

      Working out which thread (type) is the cause is often quite easy. Just run an empty(ish) thread procedure for each type -- that again does just enough by way of modifying shared data or returning plausible values to allow the program as a whole to continue to run.

      Once you've isolated the thread type involved, then apply the function-by-function mechanism as you would for a single threaded app.

      It is actually harder to describe than do.

      Good luck. (FV?)


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      RIP Neil Armstrong

        I have found that for CVs/subs Devel::Size will count entire package/class (GVs and every slice in the GV and all the shared key names of the stash and all sub stashes) that a CV is in for the size of a single sub. An XSUB with no optree comes in at a dozen KB. Be careful of what Devel::Size tells you.