Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Re: Perl and memory usage. Can it be released?

by oiskuu (Pilgrim)
on Feb 07, 2014 at 21:51 UTC ( #1073925=note: print w/ replies, xml ) Need Help??


in reply to Perl and memory usage. Can it be released?

On Linux, the glibc malloc behavior is influenced by environment variables. Large allocations are performed via mmap, smaller chunks usually live on data segment arena, which grows or shrinks via brk. Default mmap threshold might be 128k. For example:

$ strace -e brk,mmap perl -e 'pack q(x200000)'
...
brk(0x79f000)                           = 0x79f000
mmap(NULL, 200704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = ...

$ export MALLOC_MMAP_THRESHOLD_=300000
$ strace -e brk,mmap perl -e 'pack q(x200000)'
...
brk(0x79f000)                           = 0x79f000
brk(0x7db000)                           = 0x7db000
brk(0x7aa000)                           = 0x7aa000
First time, the memory was obtained via mmap; second time, by growing the arena. Arena may also shrink (here it was possible), but even if it doesn't, the unused pages are typically not much of a concern. (mmap-ed storage is unmapped when freed.)

If the process is long-lived, does great many allocations at various stages, then memory fragmentation may become a problem. (Web browsers come to mind.) When processing file after a file as you describe, this is unlikely to matter either. Memory gets allocated and released in full every time. Just be sure there are no leaks.


Comment on Re: Perl and memory usage. Can it be released?
Re^2: Perl and memory usage. Can it be released?
by bulk88 (Priest) on Feb 08, 2014 at 01:56 UTC
    POSIX brk memory will never shrink due to fragmentation. mmap memory/Win32 malloc can shrink because its all managed in a linked list chain, and the mem pages are randomly scattered through out the process.

      GNU libc allocator is derived from Doug Lea malloc, a proven general-purpose allocator. Go on, unpack and read the source and the comments (I'm looking at glibc-2.17/malloc/malloc.c)

      True, trims do not happen much because small data gets allocated from fastbins. But try to malloc a lot of somewhat larger blocks (couple hundred bytes each), and free them all. You shall see a shrink.

      Update: from said malloc.c:

      DEFAULT_TRIM_THRESHOLD 128 * 1024 DEFAULT_TOP_PAD 0 DEFAULT_MMAP_THRESHOLD 128 * 1024 DEFAULT_MMAP_MAX 65536
      And please don't say "never". E.g. freeing a block 64k to 128k in size triggers fastbin consolidation. If your program has performed a work cycle, freeing all temps, then it is quite possible a trim takes place. It depends on usage.

        /* malloc_trim(size_t pad); If possible, gives memory back to the system (via negative arguments to sbrk) if there is unused memory at the `high' end of the malloc pool. You can call this after freeing large blocks of memory to potentially reduce the system-level memory requirements of a program. However, it cannot guarantee to reduce memory. Under some allocation patterns, some large free blocks of memory will be locked between two used chunks, so they cannot be given back to the system. The `pad' argument to malloc_trim represents the amount of free trailing space to leave untrimmed. If this argument is zero, only the minimum amount of memory to maintain internal data structures will be left (one page or less). Non-zero arguments can be supplied to maintain enough trailing space to service future expected allocations without having to re-obtain memory from the system. Malloc_trim returns 1 if it actually released any memory, else 0. On systems that do not support "negative sbrks", it will always return 0. */
        You will not get a trim unless you write proof of concept code specifically to get a trim. That malloc.c says threshold for mmap is alloc size of >= 1 MB. Your process memory space will not shrink in most usage.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1073925]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (14)
As of 2014-07-25 16:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (173 votes), past polls