Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Freeing memory used by arrays

by monktim (Friar)
on Dec 04, 2003 at 14:47 UTC ( #312203=perlquestion: print w/replies, xml ) Need Help??

monktim has asked for the wisdom of the Perl Monks concerning the following question:

According to the Learning Perl book, truncating an array does not recover its memory. To feee memory I need to undef the array like this undef(@array). It also says that the memory doesn't always get free because not many operating systems support this. I find this strange as I've never had this problem in other languages.

I'm using Win2K and not much memory gets free when I undef my array. When I run the script below the @array takes ~76,000K in memory when I hit the breakpoint. After undef(@array) gets executed the memory only goes down to ~72,000K. If I continue the loop and watch memory it'll fluctuate between these two numbers.

Is there another way to free the memory? Does anyone else find this behavior strange? I must be missing something, perhaps some monks can enlighten me.

Thanks.

use strict; use warnings; print $$; my @array; my $i = 0; while (1) { $array[$i++]= '1234567890'; if ($i == 1_000_000) { undef(@array); #BREAKPOINT HERE $i = 0; } }
UPDATE
The memory is even kept after the array goes out of scope.
use strict; use warnings; print "PID: $$\n"; { my @array; my $i = 0; while ($i <= 1_000_000) { $array[$i++]= '1234567890';# for([0..1_000_000]); if ($i == 1_000_000) { undef(@array); } } } print "END\n"; # BREAKPOINT HERE, MEMORY IS STILL NOT FREE

Replies are listed 'Best First'.
Re: Freeing memory used by arrays
by Abigail-II (Bishop) on Dec 04, 2003 at 15:05 UTC
    There is a difference between "memory in use by the program" and, "memory claimed by perl from the OS, and not returned". When the program is run, the program needs memory, which is claimed from a pool managed by perl (the binary). If the pool gets empty, perl will claim more memory from the OS. If the program memory is reclaimed by the garbage collector (ref counting in the case of perl5), memory *may* be returned to the pool, or it may be kept around if perl thinks it might be reused. Perl might also return memory to the OS, but it's OS dependent whether that happens at all, and it's certainly not true that whenever the program no longer needs a piece of memory, it's returned to the OS.

    If you really need this kind of memory management, you're probably better off programming in C.

    Abigail

Re: Freeing memory used by arrays
by broquaint (Abbot) on Dec 04, 2003 at 15:02 UTC
Re: Freeing memory used by arrays
by BrowserUk (Pope) on Dec 04, 2003 at 16:56 UTC

    Perl will release memory back to the OS -- if the circumstances are right. You can see this for yourself by trying the code snippets I posted here and here.

    When your program calls for a large array to be allocated, it goes to the OS for a new chunk of memory if it doesn't already have enough available. However, the new chunk of memory isn't alocated directly to the array being created, it is allocated to perl's general memory pool. If the logic of the program then goes on to fill the new chunk of memory will the array and nothing else (like temporary variables, stack frames etc.) is allocated from that chunk of memory, then when you free the array, it is possible that it will be returned to the OS.

    However, if the logic of your program is such that the array is filled in dribs and drabs, then it is also possible that the new chunk of memory allocated will not be big enough and yet another, larger chunk my be called for. Also, if in the process of filling the array, other demands for memory (those temp vars etc.) must be satisfied, then it it is quite likely that some of the chunk will be used for that purpose as well. If those "other" sub-allocations from the big chunk requested by the OS, persist past the point in your code where you free the large array, then although the array is no longer being used, the chunk allocated for it may still contain other variables that have not yet been freed. The array's memory will be available to perl to satisfy subsequent allocations, but the chunk as a whole can not be released back to the OS until those "other" allocations have been freed. And by the time they are, some of the original array may now be reused for other purposes.

    As you can see, the picture is complicated. However, there are some steps that you can take to maximise the chance that the memory used for a large array can be released back to the OS. The major one is to pre-allocate the array to it's final size is a single step. The easiest way to do this is to assign to $#array. Eg.

    my @bigarray; $#bigarray = 1_000_000;

    However, this is not entirely without problems, nor is it the total solution. The problem with this is that once you have done it, you can no longer used some of the normal techniques for manipulating the array. If you try to push or unshift to the array having preallocated as above, then you will be extending the array, rather than using the space you've already allocated. This forces you into maintaining your own pointer into the array and explicitly assigning to the next "free" element and incrementing the pointer.

    Note: I'm not advocating anyone should do this, but if you really need to maximise the chances of memory being returned to the OS, then this is the kind of step required to do it.

    The second caveat of doing this is that when you pre-allocated the array this way, you are preallocating the space for the internal infrastructure of the array, but not whatever you subsequently store into the elements of the array! The latter will be allocated from a different chunk of memory.

    There is also an optimisation built in to perl that will cause it not to release the memory used by an array immediately, even if the programmer has apparently taken steps to indicate that the array is not longer required, if the logic of the program might indicate that it may need to be recreated at some point in the future. As an example, if you have a sub which allocates a lexically local array and then discards it when the sub exits, the memory used by that array may not be returned to perls memory pool immediately. This is because if you call the sub again, it will be more efficient to re-use the array structures and memory the second and subsequent times you call it, than to reconstruct everything from scratch. If you have a sub that creates a large array for working storage and that sub is called many times, then you will be benefiting from this optimisation without realising it.

    All-in-all, it is generally better to let perl get on with managing it's own memory allocation. Taking check points of the OS's view of a perl process memory allocation at different points in the life of that program is fraught with problems. I know, because I've (mis)spent some considerable time exploring this:)

    If you have a real need to manage the amount of memory used by your perl process, as I do for one of my pet projects that uses prodigious amounts of the stuff, then you have a long and somewhat tedious task in working out ways of achieving this. In most cases, I've found that the best way to minimize memory usage is to think hard about your algorithms and try and avoid allocating memory in the first place. There are a surprising number of ways that re-casting the obvious perl idioms allows you to manage large volumes of data without incurring the additional overhead of perls basic datatypes. These mostly revolve around loading and storing the large volumes of data in scalars instead of arrays or hashes.

    If you have a particular reason for wishing to reduce or minimize the memory usage for a particular application, then try posting (small) chunks of your code where you are using large amounts of data and request assistance in finding ways to minimize those. Best of luck!

    To pre-empt the question of "Why not use a different language than perl for those applications that use large volumes of data?". I don't want to give up access to all of perl's great features -- regexes, memory management, built-ins, OO, CPAN etc. and I don't want to have to try and re-implement these in C. I did consider, and play with, embedding a perl interpreter into a C version of my application, but it creates more problems than it cures IMO.


    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    Hooray!
    Wanted!

Re: Freeing memory used by arrays
by Ninthwave (Chaplain) on Dec 04, 2003 at 15:18 UTC
Re: Freeing memory used by arrays
by duff (Parson) on Dec 04, 2003 at 15:20 UTC
    It also says that the memory doesn't always get free because not any operating systems support this. I find this strange as I've never had this problem in other languages.

    What other programming languages have you used? Any language that implements its own memory management system (python, lisp, scheme, etc.) will exhibit similar behavior. It's only the languages like C, C++, etc where you have fine control over memory allocation/deallocation where you mayn't see this behavior. But even then, it depends on how the language is implemented. Most C implementations (for instance) won't exhibit this behavior, but as soon as you're using a C interpreter that might have some memory management scheme, all bets are off.

      I primarily use C++. Before that I primarily used C and before that PL/I. Before that I used a lot of stuff that may not have really managed memory excepting 360/370 assembler :).
      A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Freeing memory used by arrays
by Art_XIV (Hermit) on Dec 04, 2003 at 16:02 UTC

    Yet Another Explanation: Your script HAS 'freed' the memory and will make use of the 'freed' memory if it can. But your process is going to keep the memory allocated for it's own purposes unless the OS requests that it free up some.

    This is common behavior with interpreted and byte-code-based languages.

    Hanlon's Razor - "Never attribute to malice that which can be adequately explained by stupidity"
      In Java you can set an upper limit on how much memory java can consume. I wish perl had this feature.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://312203]
Approved by broquaint
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (4)
As of 2019-11-21 18:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Strict and warnings: which comes first?



    Results (105 votes). Check out past polls.

    Notices?