Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Mini-Tutorial: Perl's Memory Management

by ikegami (Pope)
on Oct 27, 2009 at 20:25 UTC ( #803515=perlmeditation: print w/ replies, xml ) Need Help??

This post answers a few common questions and misconceptions about Perl's memory management.

Lexical variables don't get freed at end of scope

Lexical variables don't get freed at end of scope. They are cleared on scope exit to give you the apperance of a fresh variable. This is an optimisation that should be transparent to you.

On scope exit, if anything prevents the variable from being resused (e.g. if there are remaining references to the variable), the existing variable becomes anonymous and a new variable is allocated.

Clearing a variable does not free the memory it uses

It was previously mentioned that variables on cleared on scope exit. Variables can also be cleared as follows:

$s = undef; @a = (); %h = ();

Clearing a variable frees no memory directly. The buffer of a string is marked as being unused, but it remains allocated for later use. The underlying array of pointers in arrays and hashes is cleared, but it remains allocated for later use. This is an optimisation that should be transparent to you.

As a result of a variable being cleared, referenced variables (incl array elements) have their refcount lowered, and that may free those variables. But not the variable being cleared.

undef does not free a variable

undef can be used to clear a variable.

undef $s; undef @a; undef %h;

undef does do more than clear the variable. It also frees any buffers the variable has.

$ perl -MDevel::Peek -e' $s = "abc"; Dump $s; $s = undef; Dump $s; undef $s; Dump $s; ' SV = PV(0x814fb00) at 0x814f69c REFCNT = 1 FLAGS = (POK,pPOK) <-- Var contains a string PV = 0x81651b8 "abc"\0 CUR = 3 LEN = 4 SV = PV(0x814fb00) at 0x814f69c REFCNT = 1 FLAGS = () <-- Var is undefined PV = 0x81651b8 "abc"\0 <-- String buffer still CUR = 3 allocated after clearing LEN = 4 SV = PV(0x814fb00) at 0x814f69c REFCNT = 1 FLAGS = () <-- Var is undefined PV = 0 <-- undef freed the buffer

This function is rarely useful.

However, undef will never free a variable. Clearing the variable may result in the reduction of the ref counts of other variables, which may free those variables. But not the variable that was passed to undef.

Freeing variables may result in no change in available system memory

You can't rely on memory being returned to the system, but it can happen.

If and when memory is returned to the OS is dependant on the memory allocation mechanism perl was compiled to use. You may also see differences between system using the same memory allocation mechanism. It's also possible that a mechanism will only ever release the large blocks it allocated.

The default memory allocation mechanism varies by OS. You are more likely to see memory being released to the OS on Windows.

Comment on Mini-Tutorial: Perl's Memory Management
Select or Download Code
Re: Mini-Tutorial: Perl's Memory Management
by EvanCarroll (Chaplain) on Oct 27, 2009 at 22:38 UTC
    Might also want to to add the reference to the perlfaq3. And, maybe mark up this post into perlfaq3. You can build perl to use your OS malloc pretty easily. I'm told Perl malloc is faster and more cross-platform, I would be curious to know why. You can find out what version your perl is using by perl -V:usemymalloc. If it is n as it is usually then you will never give back memory to the OS. I'd like to know why on Debian it is off. I assume it is the default build option, but why if gnumalloc is present?


    Evan Carroll
    The most respected person in the whole perl community.
    www.evancarroll.com

      Based on my very own experience, Perl-malloc-enabled Perl does cause the crash of xchat2(with the Perl plugin) on FreeBSD, the problem seems to be unsolved yet.

      ref: http://www.freebsd.org/cgi/query-pr.cgi?pr=121472

      I mean, sometimes, you have to usemymalloc but Perl's. :-p

        As a link: http://www.freebsd.org/cgi/query-pr.cgi?pr=121472

        If you have Perl use the system's default malloc() calls, then you can get sloppy and use Perl's configured free() on stuff that some library malloc()d or pass Perl-malloc()d buffers to an external library that expects to be able to free() or realloc() the buffer.

        So some xchat hacker got sloppy and didn't keep straight what needed to be Perl-allocated and what needed to be OS-allocated.

        - tye        

Re: Mini-Tutorial: Perl's Memory Management
by Jenda (Abbot) on Oct 27, 2009 at 22:46 UTC

    I'm not sure what is the intended audience of this article and the expected previous knowledge. As it is, it probably confuses more people than it helps.

    The fact that a variable (or rather the allocated structure) is not freed, but rather cleared and placed in some list of preallocated structures, is an implementation detail. A transparent optimization that 99.9% of people should not even know about. What they do need to know and be sure of is that the data will not be lost if there is a reference and that the memory will not be lost if there is no reference. Whether the part of memory goes to the general hash or some other place is not really their business.

    IMnsHO, there should be a big note on top that says something like "This is only for the curious, only for those that want to know what happens behind the scenes and is supposed to be hidden to the audience".

    and to tell the truth I don't understand what are you trying to say in the "undef does not free a variable" section.

    Jenda
    Enoch was right!
    Enjoy the last years of Rome.

      The fact that a variable (or rather the allocated structure) is not freed, but rather cleared and placed in some list of preallocated structures, is an implementation detail.

      I said as much. ("This is an optimisation that should be transparent to you.")

      Misconceptions about the implementation details come up all the time. You've provided me with the perfect example in this paragraph. You believe it describes Perl when it doesn't.

      Where'd you get a list of preallocated structures from? Placing blocks of memory on a list for future use is how freeing a variable usually works. This is exactly what Perl doesn't do.

      I don't understand what are you trying to say in the "undef does not free a variable" section.

      The quote line contradicts what seems to be a popular belief. The section describes what undef actually does (frees the buffers associated with the variable).

        I said as much. ("This is an optimisation that should be transparent to you.")

        That is very obtuse way

        "should" sounds a bit dangerous to me. "Should be transparent" ... or so it's not always? It can fail? Or what? How? The "should" seems to cause uncertainty.

        Mkay, let's see. If we declare the variable in a loop, we are about to do another iteration and there are no other references to the data, perl may clear the contents, but leave the structure (can't remember the name) and it doesn't have to allocate memory for that variable for the next iteration. Nice. Efficient. Now what does it do if it's not in any loop? Or it was the last iteration? What if it's a my variable inside a string eval""?

        I believe your tutorial will only make sense if you first explain at least a bit about how are the data stored (the variable name points to a SV which points to the string/contains the number/whatever) and then you can go on explaining what's "cleared" or "freed" and what does the Devel::Peek::Dump mean.

        Anyway for almost everyone the whole memory allocation tutorial should be "Declare variables with my in the tightest scope possible. Make sure you either do not create circular references, use weaken() to make sure at least one link in each circle is weak or break the circle by reseting at least one reference to undef. Do not expect perl to return the memory to the OS, but that doesn't mean the memory is not freed and will not be reused, it's just kept in perl's own hands. The rest is not your business."

        Jenda
        Enoch was right!
        Enjoy the last years of Rome.

      There are a lot of long-running perl scripts especially in the CGI area that depend on not using up more and more memory. But generally you have to do really dirty things to get bit by this. So the information is useful although probably not FAQ-worthy, especially not in this detail. And I had problems with understanding it on first reading too

      The most useful information in my opinion is "There is no guarantee that your perl version will give back memory to the system"

Re: Mini-Tutorial: Perl's Memory Management
by wfsp (Abbot) on Oct 28, 2009 at 11:26 UTC
    Lexical variables don't get freed at end of scope. They are cleared on scope exit...
    From then on in I was lost. What the difference is between "freed" and "cleared" or "end" and "exit" is in this context defeats me (and I'm not absolutely sure about lexical). Obscure in the extreme without a far deeper explanation of the subject. And that is just the second paragraph. :-)

    The title says "Mini-Tutorial: Perl's Memory Management". I would hope this is not included as or referred to as a faq/tutorial for scope or references, there are far friendly ways to do this.

      I personally think that just givin a brief preamble on what are the differences between "freed" and "cleared" as for "end" and "exit" would make the document very usefull.

        I agree, this would be much improved by some definitions. I read a lot of it thinking, "what is meant by 'free' here?".

        I think often times when questions about memory arise, a focus of interest is on when and how memory is returned to the operating system. That's probably a distinction made in here somewhere, but it would be better if that distinction were explicit.

      'free memory' is when you give up the memory block back to the OS,

      'clear memory' is when you, for instance, say @a = ();, the variable still there but without content.

      as ikegami said, perl does not free memory, so look at free in C if you still don't understand..
        for what i understood i disagree, there are 3 phases:

        clear release the bind of the data to the variable

        free the data is released to the mem manager so can be reused by perl

        free to OS may happen in some cases after the free.

        Where the first 2 are how perl behave, and the last depends on the system (OS + how perl was compiled)

        Am i wrong?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://803515]
Approved by mr_mischief
Front-paged by mr_mischief
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (15)
As of 2014-07-30 08:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (229 votes), past polls