Reference to guard not released

[roman]

Dear monks,

can anybody explain why in the following simple script, the reference to $g is kept until global destruction?

use strict;
use Scope::Guard;

my $x;
my $code = do {
    my $g;
    sub { $g = Scope::Guard->new( sub { warn "destroyed";  $x ; } ); }
+;
};

$code->();
undef $code;
warn "end";

code above yields:

end at /tmp/sample.pl line 12.
destroyed at /tmp/sample.pl line 7 during global destruction.

If I remove the reference to $x in outer scope, code behaves as expected.

use strict;
use Scope::Guard;

my $x;
my $code = do {
    my $g;
    sub { $g = Scope::Guard->new( sub { warn "destroyed"; } ); };
};

$code->();
undef $code;
warn "end";

yields:

destroyed at /tmp/sample.pl line 7.
end at /tmp/sample.pl line 12.

[Corion]

I don't find the relevant discussions and bug reports, but this feels similar to nested anonymous subroutines closing over variables. You have two nested anonymous subroutines, and the outer subroutine does not mention $x. Ideally, a workaround fix would be to just mention $x in the outer subroutine too, but that does not change the issue.

I'm sorry that I can't find the relevant links and discussions...

Potentially see also nested subs cause DESTROY block to not get hit and search for "nested subs" (but ignore all the problems with named nested subroutines).

[Athanasius]

Hello roman,

One way to get the desired behaviour is to change the argument to Scope::Guard->new() from an anonymous code reference (i.e., a closure) into a reference to a named sub:

#! perl
use strict;
use warnings;
use Scope::Guard;

my $x;
my $code = do {
    my $g;
    sub { $g = Scope::Guard->new(\&handler); };
};

$code->();
undef $code;
warn "end";

sub handler
{
    warn "destroyed";
    $x;
}

which outputs:

destroyed at /tmp/sample.pl line 18.
end at /tmp/sample.pl line 14.

as desired. I’m not sure why this is, but I suspect it has to do with the “deep magic” Perl uses to implement closures.

HTH,

[dolmen]

This is the kind of not-so-deep magic that you must understand if you use closures. Devel::RefCount may be useful to diagnose problems.

[perl-diddler]

Because you never call the sub that contains the Scope::Guard;
Try:

use strict;
use Scope::Guard;

my $x;
my $code = do {
    my $g;
    my $s=sub { $g = Scope::Guard->new( sub { warn "destroyed";  $x ; 
+} ) };
      $s->();
};

$code->();
$code=undef;
warn "end";
----
Not a CODE reference at /tmp/pt1 line 11.
destroyed at /tmp/pt1 line 7 during global destruction.

You are also creating a sub that references a 'global' ($x) that won't go out of scope until BOTH the global is 'done' and until the anonymous sub (with a reference in Scope::Guard, presumably), goes out of scope -- and it may not destroy it's reference until the program has exited (i.e. it may be in an allocated hash, once the allocated hash has been destroyed, the 'sub' goes away, and then the '$x' goes away, and everyone is destroyed...

AAiieee....

[roman]

I don't understand. Which sub I don't call? The sub which creates the guard? Of course it is called! It is stored in $code and it is called immediately after do. Your example is something very different.

The only purpose of do is to create a variable which is referenced only from the anonymous sub.

my $x;
my $code = do {
    my $g;
    sub { $g = Scope::Guard->new( sub { warn "destroyed";  $x ; 
} ) };
};
$code->();
undef $code;
warn "end";

I also don't understand the rest of your comment. The problem is not that $g references something, but that it is referenced from somewhere. But what keeps the reference to $g?

My example is the very simplification of real scenario I came across in asynchronous programming. I have two callbacks sharing a resource (handle). The resource is stored in lexical variable visible only in these two subs. First callback is called, creates the resource ($code->()) and goes out of scope. Second callback is scheduled to run. Than the process is cancelled and the second callback is cancelled from queue and goes out of scope (undef $code - here I simplified to one callback only). I expect the resource to be released then.

[dolmen]

In your example, here is what happens:
• at line 7 a closure is created that keeps references to $x and $g
• at line 10 ("$code->()"), the Scope::Guard object is instantiated and stored in variable $g, with a new closure referencing $x (or the reference of $x referenced in the closure $code).
• when $code is undef, it looks like that the closure can not be destroyed because of a circular dependency. It seems that the reference to $x kept is not the global one but the indirect one through the enclosing closure. I think this is a bug in perl.

$g is never released until global destruction. So the Scope::Guard object is never called until that time. This is one problem in your code.

The difference between when you put $x or not is that in the first case it is a closure that keeps a reference to $x while in the second case this is just a bare anonymous sub. In the second case there is no circular reference.

You should also try to call $code->() multiple times to see when the Scope::Guard object is destroyed (because the content of $g is replaced).

You can explore more what happens to references by using Devel::Refcount:

use strict;
use Scope::Guard;
use Devel::Refcount 'refcount';

sub showcount ($\$)
{
    printf "line %d: \$%s => %d\n", (caller)[2], $_[0], refcount($_[1]
+)-1;
}

my $x;

showcount(x => $x);

my $code = do {
    my $g;
    showcount(g => $g);
    # By creating a closure that reference $g, we increase its refcoun
+t to 2
    sub {
        # Here the only reference to $g is the closure
    showcount(g => $g);
    $g = Scope::Guard->new( sub { warn "destroyed"; $x; } );
    }
    # Here $g is out of scope, so refcount of $g decreases to 1
};

showcount(x => $x);
$code->();
showcount(x => $x);
#$code->();
showcount(x => $x);
undef $code;
# Here, I expect the refcount of $x to decrease, but it doesn't
showcount(x => $x);
warn "end";

[dolmen]

Well, it is more interesting to show when the closure stored in $code is effectively destroyed:

use strict;
use Scope::Guard;

my $x;

my $code = do {
    my $g;
    sub {
        my $z;
        $g = Scope::Guard->new( sub { warn "destroyed"; $z } );
    }
};

use Scalar::Util ();
Scalar::Util::weaken(my $code2 = $code);

$code->();
printf "code2: %d\n", !!(defined $code2);
undef $code;
printf "code2: %d\n", !!(defined $code2);
warn "end";

Output on perl 5.14.2:

code2: 1
code2: 1
end at x.pl line 21.
destroyed at x.pl line 10 during global destruction.

As shown above, the issue is independent of $x being global: here I'm using $z which is local to the enclosing closure and the behavior is the same: the closure pointed by $code is not freed.

[andal]

I can't explain what is happening, but I can offer work around :)

use strict;
use Scope::Guard;

my $x;
my $code = do {
    my $tmp = sub { warn "destroyed";  $x ; };
    my $g;
    sub { $g = Scope::Guard->new( $tmp ); } ;
};

$code->();
undef $code;
warn "end";

[sundialsvc4]

You created a block of code, called a “closure,” which references a variable outside itself and which therefore seizes a reference-count to that data object.   The subroutine itself is never called, but it nevertheless exists. It is built at compile time.   Even though it is dead-code, Perl generates it anyway.

You can confirm my assertion that the sub is never called by this one-liner:
perl -e 'sub { print "foo!\n"; }'
... which produces no output.

[roman]

Which subroutine is never called? The one which is the last command in do block and which is thus stored in $code? It is called explicitly right after do block - (see $code->())

my $code = do {
    my $g;
    sub { $g = Scope::Guard->new( sub { warn "destroyed";  $x ; } ); }
};

$code->();

[perl-diddler]

So what is ScopeGuard supposed to be guarding? "$x"?

You declared $x at the global level. Is that what you are protecting? That doesn't get destroyed until the program exits...

You declared an anonymous sub inside scopeguardnew... that's also unlikely to be destroyed until your program exits -- because Scope::Guard is holding a "reference" to the interior sub (likely)... and, AT THE LEAST, the 'my $x' is declared at the global level -- so it won't be destroyed until the prog exits...

But I checked, and putting braces around the dcl of $x, doesn't change the output. So I would bet scopeguard has stored a reference to the interior sub so it can be called when "$x" goes out of scope.

But you referenced "$x" in side the sub that scope guard stored...--- so $x can't go out of scope until the interior sub (that you passed to SG->new) disappears -- and it won't disappear until your prog exits...

All you did by assigning undef to $code is lose the reference to the outside sub.. SGnew still has a reference to the inner sub and $x...

The reference to $x, that is in the anonymous sub, is referred to in jargon, as a "closure".

I'd guess that's because it creates a closed "ecosystem" that holds the anon-sub and the anon-sub's copy of $x -- no one changing '$x', after that sub is defined, will affect the value that is stored (referenced, really), in that inner sub (inside the new)...

Does that make more sense?