Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: Re: Memory usage and perl

by fokat (Deacon)
on Mar 14, 2003 at 16:16 UTC ( #243094=note: print w/replies, xml ) Need Help??

in reply to Re: Memory usage and perl
in thread Memory usage and perl

What broquaint and robartes are trying to say is that the "memory pools" used by Perl are allocated in virtual memory while only what is known as the working set of memory pages is actually mapped into physical memory. Let me explain this a bit better...

In *nix and in general, in any virtual memory-enabled OS I know, processes are allocated a chunk of virtual memory. This refers to memory that is known to belong to this process (and perhaps shared by others as well, but this is not relevant now).

The pages of memory that compose this virtual memory need to be stored somewhere. So there are two places to do so: Your computer's memory (also called in core because old memory technologies used magnetic cores to store ones and zeros) and a secondary device such as a disk (also called paged-out or swapped memory).

To accomplish this feat, the OS allocates some chunk of disk for this purpose. Some do that on demand, some do that as a pre-allocation but this is not too important for this matter. The fact is that seldom used memory ends up in a disk, not in your core memory. So, you can have a machine with say, 128M of RAM and a 2 Gb swap partition. Your Perl process could probably achieve 600M of virtual size, but have a much smaller working set (say, 60M), so this is the only chunk of its address space that ends up mapped in the core memory.

None of the OSes I know give the process the capability of returning unused memory, so as broquaint said, this is an OS issue and not a Perl issue.

Of course, all this is from the OS perspective. Perl's memory management have a more detailed view of what's going on, and simply marks chunks of memory as used / unused, reusing the previously unused memory blocks to keep from requesting more from the OS. This is a common strategy that works well in 99% of the practical scenarios you will find.

I commonly run Perl tasks that can take up to 900M of virtual space, yet span a working set of a few tens of megabytes. If you add up the virtual spaces of a couple of them, you far exceed the real memory on the machine where this runs, so as you see, this is not a real problem. However, if your program lacks a property called locality of reference, this whole scheme will work against you. Your only solution there, short of rethinking your algorythm, is adding more real memory to your machine.

Hope this long clarification is of some help...

Best regards

-lem, but some call me fokat

Replies are listed 'Best First'.
Re: Re: Re: Memory usage and perl
by derby (Abbot) on Mar 14, 2003 at 16:25 UTC
    fokat++ ... but just to nitpick mmap and certain mallocs (and under certain conditions) will return the memory to the system. Check out the Unix Programming FAQ section 1.12.


      Thanks a lot for the reference, derby (and ++ for your nitpicking). I have a small difference, though...

      Actually, mmap() does not allocate memory. What it does is map pages belonging to a file, probably from the buffer cache, into your process' private address space. Certainly it is a neat trick, but might have undesired side effects when competing with file-related resources in the same machine, specially with a static buffer cache.

      They will "return" the memory to the OS because the memory was not originally allocated to your process. I am glad to see certain implementations of malloc() and free() taking advantage of this.

      Best regards

      -lem, but some call me fokat

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://243094]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (9)
As of 2017-09-25 11:57 GMT
Find Nodes?
    Voting Booth?
    During the recent solar eclipse, I:

    Results (280 votes). Check out past polls.