http://www.perlmonks.org?node_id=95940

tachyon has asked for the wisdom of the Perl Monks concerning the following question:

I came across an unusual form of closure that I would like someone to explain to me. You make a regular closere like this:

{ my $x; sub foo { return ++$x; } } for (1..7) { print foo() }

This prints 1234567 because Perl maintains the $x variable as sub foo's private memory. OK no suprises there. It's a closure.

Why does this code produce the same effect?

sub foo { my $x if undef; return ++$x; } for (1..7) { print foo() }

It must be a closure but why does Perl maintain the $x var. If you remove the if undef $x behaves as a lexically scoped var, so as expected we print 1111111 as $x is destroyed and recreated every time you call the sub. However with this added it forms a closure so each time we call foo() $x increments and we print 1234567. A bit of experimentation shows any false value 0, '', or undef elicits this behaviour. Could someone please explain this?

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Replies are listed 'Best First'.
Re: Unusual Closure Behaviour
by Abigail (Deacon) on Jul 12, 2001 at 11:47 UTC
    The jury is still debating whether this is a feature or a bug. All I can offer is an explanation.

    my has compile-time *and* run-time effects. At compile time, the compiler knows about the variable, etc. At run-time, values are set, my $x; makes that $x becomes undef. So far, so good.

    However, for efficiency reasons, if Perl exits a block, it will actually *not* delete any variables lexical to the block. You cannot refer to them anymore (the compiler takes care of that), but the data structure build for it remains. Perl does this because it is likely that you reenter a block and if the structure remains, Perl can save time rebuilding it. However, with my $x if undef, no run-time effect on $x happens when reentering the block. (The first time the block is entered, the datastructure gets build when $x is used). And since the structure doesn't get rebuild, the value doesn't get reset either. So, you have created a static variable....

    -- Abigail

      It is a bug. As it is written in the tractate Camel III 4:132:
      Lexicals are hidden from any subroutine called from their scope. This is true even if the same subroutine is called from itself or elsewhere—each instance of the subroutine gets its own "scratchpad" of lexical variables.
      $x is clearly a lexical. Its scope is inside foo. Yet each instance of foo is not getting its own scratchpad of lexical variables. Hence, we have a creepy crawly thing of uncertain appeal...a bug.

      Update 2: One might argue that the anomalous behavior of $x is consistent with the above dictum, because the subroutine foo is not being called from scope of $x. However, when the Perl wise ones write that lexicals are hidden from any subroutine called from their scope, it means even if they are called from their scope.

      If a lexical is hidden from subroutines called from inside its own scope, then how much more so must it be hidden from subroutines called from outside its own scope.

      As Abigail has taught, conditionally declaring lexicals for use in subroutines is an example of "an ox that is known to gore," i.e., its behavior is going to be ambiguous. Even though it is legal to release such an ox and let it run around and possibly damage a neighbor's code, let us instead "build a fence around the Law" and only declare lexical variables in subroutines unconditionally, outside of conditional branching such as 'my $x if 0'.

      (previous updates effaced)

        Well, technically what is happening is not violating Camel III. See, it *does* get its own scratchpad, which can be quickly checked by:
        #!/opt/perl/bin/perl -w use strict; sub foo; sub foo { return unless $_ [0]; my $x if 0; print ++ $x, " "; foo $_ [0] - 1; } foo 1; print "\n"; foo 2; print "\n"; foo 3; print "\n"; foo 4; print "\n"; __END__ 1 2 1 3 2 1 4 3 2 1
        Scary, isn't? When the subroutine recurses, it notices there is still a reference to $x and hence it will create a new scratchpad. But when there is no reference, it will reuse an old scratchpad....

        I do agree that using my in this way doesn't tend to lead to well understood code. I wouldn't go as far as to say it's never useful, but such cases will be very rare and it's not a technique I would teach in my classes.

        -- Abigail

(tye)Re: Unusual Closure Behaviour
by tye (Sage) on Jul 12, 2001 at 14:27 UTC

    First, neither of those are closures. They are both examples of static variables and named functions. The term "closure" refers to a code reference that also carries along some variables. In neither of these case is the $x being carried around in a code reference (in part because no code references are being used).

    Second, I (if anyone cares) don't consider my $x if 0; to be a good thing to actually use. A while back I had an idea to allow BEGIN to be a statement modifier just like for, while, if, etc. Then you could implement static variables like:

    sub foo { my $x= 'a' BEGIN; return ++$x; }
    which has the benefit of allowing you to initialize your static vars and of cluing you into the fact that the initialization happens at compile time so you can't use anything that isn't available yet.

    This feature would have other nifty uses. For example: my $haveCGI= eval { require CGI }   BEGIN; But the one that could be really fun would be: eval "my $ARGV[0];"   BEGIN; if you could stop eval from implying an enclosing scope. This could be very powerful and give TheDamian lots of new ways to do really scary things in Perl. (:

            - tye (but my friends call me "Tye")
      A while back I'd talked to Randal about closures, and he told me that a closure needn't be a code reference:
      >>>>> "Jeff" == Jeff Pinyan <jeffp@crusoe.net> writes: Jeff> A closure is an ANONYMOUS function (constructed via $x = sub { Jeff> ... }) that contains LEXICAL variables that have been defined in Jeff> a scope visible to the closure itself. leave out the word ANONYMOUS there. ANONYMOUS and CLOSURE are orthogonal. in "BEGIN { my $x; sub foo { ... $x ... } }", foo is a CLOSURE and is not ANONYMOUS.


      japhy -- Perl and Regex Hacker

        That is an interesting case BEGIN { my $x; sub foo { ... $x ... } } Perl could decide to implement that as a closure (and probably does because "BEGIN" blocks are implemented as subroutines). But lets find out:

        #!/usr/bin/perl -w use strict; use Devel::Peek qw(Dump); BEGIN { my $x; sub begin { ++$x; } } sub justmy { my $x; ++$x; } sub ifmy { my $x if 0; ++$x; } { my $x; sub static { ++$x; } } sub nest { my $x; sub inner { ++$x } } sub gen { my $x; return sub { ++$x }; } *insert= gen(); Dump $_ for( \&begin, \&justmy, \&ifmy, \&static, \&inner, gen(), \&insert + );
        The "cleaned up" output is:
        Variable "$x" will not stay shared at closure.pl line 27. begin: SV = RV(0x1a83a20) at 0x1a65068 SV = PVCV(0x1a8340c) at 0x1a62144 GVGV::GV = 0x1a7b6c0 "main" :: "begin" PADLIST = 0x1a7b690 1. 0x1a621b0 (FAKE "$x" 0-57) OUTSIDE = 0x1a620fc (UNIQUE) justmy: SV = RV(0x1a83a2c) at 0x1a65098 SV = PVCV(0x1a8345c) at 0x1a620f0 GVGV::GV = 0x1a7b72c "main" :: "justmy" PADLIST = 0x1a7b708 1. 0x1a7b714 ("$x" 59-60) OUTSIDE = 0x1a6f124 (MAIN) ifmy: SV = RV(0x1a83a30) at 0x1a650b0 SV = PVCV(0x1a7adbc) at 0x1a7b750 GVGV::GV = 0x1a7b798 "main" :: "ifmy" PADLIST = 0x1a7b774 1. 0x1a7b780 ("$x" 61-62) OUTSIDE = 0x1a6f124 (MAIN) static: SV = RV(0x1a839dc) at 0x1a650e0 SV = PVCV(0x1a7ada4) at 0x1a7b76c GVGV::GV = 0x1a7b7a8 "main" :: "static" PADLIST = 0x1a7b790 1. 0x1a620f0 (FAKE "$x" 0-64) OUTSIDE = 0x1a6f124 (MAIN) inner: SV = RV(0x1a83a34) at 0x1a650c8 SV = PVCV(0x1a7ae5c) at 0x1a7b81c GVGV::GV = 0x1a7b858 "main" :: "inner" PADLIST = 0x1a7b840 1. 0x1a7b7ec (FAKE "$x" 0-64) OUTSIDE = 0x1a7b7bc (nest) gen(): SV = RV(0x1a83a38) at 0x1a65158 SV = PVCV(0x1a7af9c) at 0x1a650e0 FLAGS = (ANON,CLONED) GVGV::GV = 0x1a7b924 "main" :: "__ANON__" PADLIST = 0x1a65128 1. 0x1a65080 (FAKE "$x" 0-67) OUTSIDE = 0x1a7b894 (gen) SV = PVCV(0x1a7aeac) at 0x1a7b894 GVGV::GV = 0x1a7b93c "main" :: "gen" PADLIST = 0x1a7b8b8 1. 0x1a65170 ("$x" 66-68) 2. 0x1a7b8e8 ("&" 1--1) OUTSIDE = 0x1a6f124 (MAIN) insert: SV = RV(0x1a83a3c) at 0x1a65188 SV = PVCV(0x1a7af4c) at 0x1a6f01c FLAGS = (ANON,CLONED) GVGV::GV = 0x1a7b924 "main" :: "__ANON__" FLAGS = 0x6 PADLIST = 0x1a6f0f4 1. 0x1a7b8c4 (FAKE "$x" 0-67) OUTSIDE = 0x1a7b894 (gen) SV = PVCV(0x1a7aeac) at 0x1a7b894 GVGV::GV = 0x1a7b93c "main" :: "gen" PADLIST = 0x1a7b8b8 1. 0x1a65170 ("$x" 66-68) 2. 0x1a7b8e8 ("&" 1--1) OUTSIDE = 0x1a6f124 (MAIN)
        or just consider
        begin: 1. 0x1a621b0 (FAKE "$x" 0-57) justmy: 1. 0x1a7b714 ("$x" 59-60) ifmy: 1. 0x1a7b780 ("$x" 61-62) static: 1. 0x1a620f0 (FAKE "$x" 0-64) inner: 1. 0x1a7b7ec (FAKE "$x" 0-64) gen(): 1. 0x1a65080 (FAKE "$x" 0-67) insert: 1. 0x1a7b8c4 (FAKE "$x" 0-67)
        which seems to indicates that "justmy" and "ifmy" are not implemented as closures but all of the rest are implemented as closures. So I'll certainly be more lenient in what I let other people call closures. (:

        But it also indicates that the padlist is carried around for ordinary subroutines, which makes that aspect of the implementation less important to me.

        I think that the important thing about closures is being able to call the same code but have it use different variables (without passing them in as arguments). Above, only the anonymous subroutine and "insert" meet that criterion. So those are what I'll call closures. The other 3 cases I'll call "static variables that Perl implements via closures" if pushed. :)

                - tye (but my friends call me "Tye")
        The docs are pretty clear that using sub to create a ref to an anonomous sub will do closures and that a normal named sub will not.

        So I tried it. It gives me a warning that "$x will not stay shared", but the result seems to work! That is, a created named function seems to reference the same variable as a standard closure created in the same scope, and running the creator again (which makes a different local $x) keeps distinct identities.

        So what's going on here? Are the docs outdated? Is this working by accident or happenstance? Does the presence of a regular closure somehow make it work?

        —John

        use v5.6.1; # Active State build 626 use strict; use warnings; sub outer { my $x= shift; my $name= shift; my $closure= sub { return $x++; }; eval "sub $name { return \$x++; }"; return $closure; } my $r1= outer (1, 'f1'); my $r2= \&f1; print $r1->(), $r2->(), "\n"; my $r3= outer ('A', 'f2'); my $r4= \&f2; print $r3->(), $r4->(),$r3->(), $r4->(), "\n"; print $r1->(), $r2->(),$r1->(), $r2->(), "\n";
      his first example
      { my $x; sub foo { return ++$x; } }
      is very much a closure, because $x is lexically scoped to that block, and sub foo is by definition global and survives the loop, $x goes away, except for the one reference in sub foo. his second one, however, would not be a closure.

                      - Ant

      Says tye:
      First, neither of those are closures. They are both examples of static variables and named functions. The term "closure" refers to a code reference that also carries along some variables. In neither of these case is the $x being carried around in a code reference (in part because no code references are being used).
      There is indeed code reference. It is stored in the symbol table for the main package. It points to a CV, just like any other code reference, and the CV has a pointer to the pad, which does have the $x variable captured in it, just like any other closure.

      So yes, it is a closure.

      --
      Mark Dominus
      Perl Paraphernalia

Re: Unusual Closure Behaviour
by MeowChow (Vicar) on Jul 12, 2001 at 11:06 UTC
    It's a feature (cough, cough...)
       MeowChow                                   
                   s aamecha.s a..a\u$&owag.print

      Ah so this is one of those famous Perl unintentional features?

      cheers

      tachyon

      s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Re: Unusual Closure Behaviour
by synapse0 (Pilgrim) on Jul 12, 2001 at 11:45 UTC
    My thoughts, though possibly ignorant:
    It seems like it's doing (almost) what it's supposed to, since you are using my on a false condition, it doesn't "reset" $x..
    But why doesn't it complain about global $x if strict is on? Well, my initial thoughts fall back on remembering hearing something about my working compile time, but that's fuzzy at best.
    I'm not really sure, but i believe it's doing (almost) what it's supposed to.
    -Syn0
Re: Unusual Closure Behaviour
by Dominus (Parson) on Jul 14, 2001 at 19:10 UTC
What exactly is a closure?
by John M. Dlugosz (Monsignor) on Jul 12, 2001 at 22:51 UTC
    According to the perlref page, only anonymous subs are closures. Examples elsewhere show that named subs don't "nest" the same way.

    However, my experiments indicate that they work as closures, the way I did it.

    So, I think the real difference is that just declaring a nested sub creates the sub at compile time. An anonymous sub is created at run time. My using eval to make the named sub appear at run time accomplished the same thing.

    So, I think the real distinction is: A sub is a closure if it is created after the run-time behavior of my has occured. A sub that is formed after my is seen at compile-time can refer to that variable, but does not act as a closure. Unless the block containing that lexical variable is executed more than once, that doesn't matter.

    The Perl compiler is coded to warn about named subs, but that is not the condition is should be checking for. It warns about cases that do work correctly, because the real requirement is that the sub be formed after the my is run.

    A named sub can become a closure if it is delayed using eval, and an anonymous sub can be un-closed if it is hastened using BEGIN.

    Could someone familiar with perlguts and the actual internal trees confirm or clarify this?

    —John

Re: Unusual Closure Behaviour
by BMaximus (Chaplain) on Jul 12, 2001 at 15:54 UTC
    $x should dissapear when it leaves the scope of foo. Why isn't this happening? Seems like the table is keeping it. If you take out the if undef the variable keeps on being redefined and reset. Could be useful for recursion :)

    BMaximus
      $x should not disappear outside the scope of foo. my $x if undef; means that $x is not local; the my, though noted at compile time (mainly to prevent strict complaints), does not actually produce a scoping effect until run-time (if I understand the perldocs correctly). Therefore, the my $x is never evaluated for scoping purposes, since undef is always false.
        HyperZonk and Synapse0 have both suggested that the my is being ignored and all that is happenning is that $x is simply global.

        It's a good theory, and certainly what I thought when I first looked at it. But it's easy to see that this isn't true: try comparing

        sub foo{ my $x if 0; print "FOO: ", $x++, "\n"; } sub bar{ my $x if 0; print "BAR: ", $x--, "\n"; } foo, bar for (1..10);
        with
        sub foo{ print "FOO: ", $x++, "\n"; } sub bar{ print "BAR: ", $x--, "\n"; } foo, bar for (1..10);

        I reckon the real reason is the explanation given by Abigail.

        Now I don't think I would ever write my $x if 0 unless I wanted to get the sack for writing obscure code, but how about:

        sub STATIC_VARIABLE{ 0 } sub foo{ STATIC_VARIABLE && my $x; print "FOO: ", $x++, "\n"; }
        would that be difficult to understand? Would it carry on working in the future?

        -- iakobski

Re: Unusual Closure Behaviour
by paulbort (Hermit) on Jul 12, 2001 at 21:08 UTC
    The word that I keep seeing left out of these replies is 'precedence'. It looks to me like there is confusion between my ($x if undef); and (my $x) if undef; It looks to me like you are expecting the latter and getting the former. I suspect 'if' is being evaluated as an operator, so it gets a higher precedence than the 'my' function, and confusion results. Or I could be completely wrong.

      Hi I think this clarifies the situation. $x is definitely lexically scoped. Run this as in then uncomment the use strict.

      # use strict; sub foo { my $x if 0; return ++$x; } for (1..7){ print foo(); print "($x)"; };
      cheers

      tachyon

      s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

      Actually, the if being evaluated first is exactly what we would want in any case. You are missing the implication of the behavior, however. The variable, besides being localized which is unexpected (until one realizes that data scoping is done at compile-time in this case), also becomes static; that is, it retains its value between calls, which is unexpected behavior for a local via my variable. See iakobski's reply in this thread for more details.
Re: Unusual Closure Behaviour
by artist (Parson) on Jul 12, 2001 at 20:12 UTC
    HI, Try this, Here it says that $x is not defined and therefore it creates a new local variable $x with 'my'.
    Otherwise it uses the last value of x which is the default behaviour.
    sub foo { my $x if !defined $x; return ++$x; } for (1..7){ print foo(); };

      Actually, that code isn't doing what you think. One way to see is to just add "use strict" to it. You'll get: Global symbol "$x" requires explicit package name

      A more fun way is:

      sub foo { my $x= $_[0] if ! defined $x; print "(@_):", ++$x, " "; undef $x if @_ && shift; } foo($_) for( 0,0,5,4,3,0,5,0,4,0,3 ); print $/, $x= 10, $/; foo($_) for( 0,0,5,4,3,0,5,0,4,0,3 ); print $/;
      which produces:
      (0):1 (0):1 (5):6 (4):5 (3):4 (0):1 (5):6 (0):1 (4):5 (0):1 (3):4 10 (0):1 (0):2 (5):3 (4):1 (3):1 (0):1 (5):2 (0):1 (4):2 (0):1 (3):2
      The $x in "if !defined $x" is the global $main::x because my variables can't be used until after the end of the statement in which they were declared.

      So your code is always initializing the lexical $x in your subroutine because you have never defined the global $main::x that you are checking against.

              - tye (but my friends call me "Tye")
Re: Unusual Closure Behaviour
by RMGir (Prior) on Feb 25, 2004 at 12:48 UTC
    (Refresher: tachyon was asking about sub foo { my $x if undef; return $x++;})

    It looks like the bug/feature question is finally settled, solidly on the "bug" side.

    This p5p summary refers to this thread where Dave Mitchell submits a patch to make it warn deprecated, and exposes a lot of odd code in the process...


    Mike