Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

What broquaint and robartes are trying to say is that the "memory pools" used by Perl are allocated in virtual memory while only what is known as the working set of memory pages is actually mapped into physical memory. Let me explain this a bit better...

In *nix and in general, in any virtual memory-enabled OS I know, processes are allocated a chunk of virtual memory. This refers to memory that is known to belong to this process (and perhaps shared by others as well, but this is not relevant now).

The pages of memory that compose this virtual memory need to be stored somewhere. So there are two places to do so: Your computer's memory (also called in core because old memory technologies used magnetic cores to store ones and zeros) and a secondary device such as a disk (also called paged-out or swapped memory).

To accomplish this feat, the OS allocates some chunk of disk for this purpose. Some do that on demand, some do that as a pre-allocation but this is not too important for this matter. The fact is that seldom used memory ends up in a disk, not in your core memory. So, you can have a machine with say, 128M of RAM and a 2 Gb swap partition. Your Perl process could probably achieve 600M of virtual size, but have a much smaller working set (say, 60M), so this is the only chunk of its address space that ends up mapped in the core memory.

None of the OSes I know give the process the capability of returning unused memory, so as broquaint said, this is an OS issue and not a Perl issue.

Of course, all this is from the OS perspective. Perl's memory management have a more detailed view of what's going on, and simply marks chunks of memory as used / unused, reusing the previously unused memory blocks to keep from requesting more from the OS. This is a common strategy that works well in 99% of the practical scenarios you will find.

I commonly run Perl tasks that can take up to 900M of virtual space, yet span a working set of a few tens of megabytes. If you add up the virtual spaces of a couple of them, you far exceed the real memory on the machine where this runs, so as you see, this is not a real problem. However, if your program lacks a property called locality of reference, this whole scheme will work against you. Your only solution there, short of rethinking your algorythm, is adding more real memory to your machine.

Hope this long clarification is of some help...

Best regards

-lem, but some call me fokat

In reply to Re: Re: Memory usage and perl by fokat
in thread Memory usage and perl by Anonymous Monk

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others musing on the Monastery: (5)
    As of 2018-04-27 09:28 GMT
    Find Nodes?
      Voting Booth?