Not able to release memory

[asinghvi]

Perl gurus,
I am trying to address some memory limitation in my perl program. While doing my research, I was curious to see how memory gets released in perl once a variable goes out of scope.
I have the following code:

#!/usr/bin/perl
undef %hash1;
foreach ($i=1;$i<=1000;$i++) {
   foreach ($j=1;$j<=100;$j++) {
      foreach ($k=1;$k<=100;$k++) {
         $hash1{"$i^$j^$ik"} = 1;
  }
 }
}
print "Check Memory usage NOW\n";
sleep 10;
undef %hash1;
print "And Now\n";
sleep 10;

I see that when the data is loaded, the memory of this process is (e.g. 200MB). But after destructing "%hash1", I don't see any change in the perl process/residual memory. Isn't it supposed to reduce the process size of the perl process.
Am I missing something here

Thanks in advance for your help
-A

[davido]

Perl may be done using that memory for the moment, but virtually all OS's don't reclaim it. However, if you were to rebuild another data-structure that needed gobs of memory, the same memory would be reused within your current process.

Update: I should mention that if you really need to force your OS to reclaim memory, you should start looking at using exec to spawn a new process while terminating the current one. Of course you may need to figure out a solution to maintain state, but that's another discussion.

[waswas-fng]

++ davido, Some of the reasons that most OS will not reclaim memory very much or at all until the process exits are: Fragmentation -- are the small chunks of memory released valuable to other applications? If so does the OS have to restructure and virtualize the memory so that the free blocks are contiguous? At what cost?

Unix processes tend to live in two forms, quick running (ls, cat, du) and long running (syslog, inetd, oracle). For quick running processes the process will be over soon enough and the memory free issue is moot. For long running processes the process will generally need to alloc more memory anyways, may as well make it easy and quick by leaving it for the next alloc. This happens at little expense to your physical RAM as while that memory is freed the chunk can be swapped out and used by other applications.

You will see that on most modern unices that large block (one alloc, one free) tend to be reabsorbed by the OS -- although for instance on Solaris with an app compiled with gcc you will not see this. it all depends on the OS, the compiler and the libraries used. Not really a perl issue.

[Anonymous Monk]

In my case, I have Tk application running in Win32 environment that never exits. Some of my variables get very large data and as a result - after a period of time, my application "eats" the memory of Windows, and windows complain that the virtual memory is too low. Most of the time - as a result of this - the application gets destroyed by Windows. What I recently found is that if one click with the mouse on the Minimize button of the Tk window, Windows is claiming back the unused memory from perl and everything goes back to normal. For example: from 128MB wperl.exe in the memory, the process size goes to 2MB. Then it starts growing again - until next time you minimize the window. So what I was thinking is that there must be an API call that identifies the process and ask the OS to require the garbige. Correct me if I am wrong. I saw this happens with many applications in Windows. I am desperate for solution about this - because I need to finish my project :(( any help is welcome. Thanks!!!

[eyepopslikeamosquito]

A while back, I did a simple test of a C program mallocing a 4 MB chunk of memory, then freeing it. The memory was returned to the OS on Windows and Linux -- but not on every other of the 12 different (mostly Unix-derived) platforms I ran the test on.

Whether the memory is returned to the OS or not depends on the implementation of the user-level malloc function. Traditional Unix implementations of malloc simply call sbrk(2) to increase the address space of the process. Since this call is expensive, when you call free, they typically do not return the memory to the OS, but keep it to reuse next time you call malloc.

glibc 2.x based Linux systems use ptmalloc, which is based on Doug Lea's malloc. This version of malloc is able to return memory to the OS because it uses mmap(2), rather than sbrk, when the chunk of memory requested is larger than some threshold value (typically 128K).

MSVC CRT malloc similarly uses a "small block" and "large block" heap heuristic -- but uses Win32 heaps, rather than sbrk/mmap, under the covers.

So, if you were desperate to have memory returned to the OS, you could go to the bother of building a custom perl which used a different implementation of malloc.

[demerphq]

Ok, there are a few things.

First as others have commented the OS manages allocating and freeing memory, and typically doesnt bother. But with modern OS'es swapping etc, im not sure why it should be a problem, maybe on *nix its different to win32.

Second you are creating 1000 * 100 * 100 keys, which means 10,000,000 keys, which also means a hash stucture of 16,777,216 value slots as internally there will always be next power of 2 value slots vs keys. The best way to reduce the memory footprint of something like this is to use a DB or a hash tied to a DB of some form, possibly cached by a real hash for speed on frequent fetches. A tied DB_File B+Tree is often a reasonable option.

Third your style is ah, very C like. :-)

#!/usr/bin/perl -w
use strict;
my %hash;
for my $i (1..1000) {
    for my $j (1..100) {
        for my $k (1..100) {
            $hash{$i,$j,$k} = 1
        }
    }
}
print "Check Memory usage NOW and press enter\n";
my $chk=<>;
%hash=();
print "Check Memory usage NOW and press enter\n";
$chk=<>;

HTH

[kvale]

In most Unices, process memory is not 'given back' to the OS until the process finishes.

[eyepopslikeamosquito]

Just a couple of random extra thoughts that may be of interest. To find the memory usage of Perl variables, try the Devel::Size module (by a very distinguished and busy Perl-6 architect, I see;-) and Devel::Peek.

To pre-size a hash, use the keys function as a l-value; see perldoc -f keys for details. That is, if you know in advance that the hash is going to be very big, pre-sizing it will certainly improve speed, and may improve memory usage a little (untested).

[Roger]

Not possible with standard ActiveState Perl under Windows. However Perl can be compiled from source to use custom malloc/dealloc/realloc functions, provided by the Windows operating system (you will probably need to write a wrapper for it), instead of using Perl's own. Which will almost certainly suffer in performance, but will be more memory friendly.