Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Memory usage and perl

by Hena (Friar)
on Mar 14, 2003 at 15:16 UTC ( #243077=note: print w/ replies, xml ) Need Help??


in reply to Memory usage and perl

I posted this, but forgot to login. So...

Anyways i checked the other thread and to note i'm using RH with perl-5.6.1-34.99.6 rpm. So different OS.

Well there was also mentioned garbage collector and memory pool for perl. So there is no way of clearing this "pool" from perl. Meaning if i have a perl script running longer time and at some point it takes up a lot of memory it will also keep this same memory until perl process itself is exited?


Comment on Re: Memory usage and perl
Re: Re: Memory usage and perl
by fokat (Deacon) on Mar 14, 2003 at 16:16 UTC

    What broquaint and robartes are trying to say is that the "memory pools" used by Perl are allocated in virtual memory while only what is known as the working set of memory pages is actually mapped into physical memory. Let me explain this a bit better...

    In *nix and in general, in any virtual memory-enabled OS I know, processes are allocated a chunk of virtual memory. This refers to memory that is known to belong to this process (and perhaps shared by others as well, but this is not relevant now).

    The pages of memory that compose this virtual memory need to be stored somewhere. So there are two places to do so: Your computer's memory (also called in core because old memory technologies used magnetic cores to store ones and zeros) and a secondary device such as a disk (also called paged-out or swapped memory).

    To accomplish this feat, the OS allocates some chunk of disk for this purpose. Some do that on demand, some do that as a pre-allocation but this is not too important for this matter. The fact is that seldom used memory ends up in a disk, not in your core memory. So, you can have a machine with say, 128M of RAM and a 2 Gb swap partition. Your Perl process could probably achieve 600M of virtual size, but have a much smaller working set (say, 60M), so this is the only chunk of its address space that ends up mapped in the core memory.

    None of the OSes I know give the process the capability of returning unused memory, so as broquaint said, this is an OS issue and not a Perl issue.

    Of course, all this is from the OS perspective. Perl's memory management have a more detailed view of what's going on, and simply marks chunks of memory as used / unused, reusing the previously unused memory blocks to keep from requesting more from the OS. This is a common strategy that works well in 99% of the practical scenarios you will find.

    I commonly run Perl tasks that can take up to 900M of virtual space, yet span a working set of a few tens of megabytes. If you add up the virtual spaces of a couple of them, you far exceed the real memory on the machine where this runs, so as you see, this is not a real problem. However, if your program lacks a property called locality of reference, this whole scheme will work against you. Your only solution there, short of rethinking your algorythm, is adding more real memory to your machine.

    Hope this long clarification is of some help...

    Best regards

    -lem, but some call me fokat

      fokat++ ... but just to nitpick mmap and certain mallocs (and under certain conditions) will return the memory to the system. Check out the Unix Programming FAQ section 1.12.

      -derby

        Thanks a lot for the reference, derby (and ++ for your nitpicking). I have a small difference, though...

        Actually, mmap() does not allocate memory. What it does is map pages belonging to a file, probably from the buffer cache, into your process' private address space. Certainly it is a neat trick, but might have undesired side effects when competing with file-related resources in the same machine, specially with a static buffer cache.

        They will "return" the memory to the OS because the memory was not originally allocated to your process. I am glad to see certain implementations of malloc() and free() taking advantage of this.

        Best regards

        -lem, but some call me fokat

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://243077]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (9)
As of 2014-11-28 22:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (200 votes), past polls