Autovivification with hash of hashes

[Mr_Person]

Hi, I'm writing a program that uses a hash of hashes. At one point this program checks to see if a hash exists within a hash. Much to my suprise, checking for its existence caused that key to come into existence. Below is some sample code that illustrates what I'm talking about:

#!/usr/bin/perl -w

use strict;
use warnings;

my %foo;

if(exists($foo{qux})) {
    print "\$foo{qux} exists";
}

if(exists($foo{bar}{baz})) {
    print "\$foo{bar}{baz} exists\n";
}

if(exists($foo{bar})) {
    print "\$foo{bar} popped into existence\n";
}

if(exists($foo{qux})) {
    print "\$foo{qux} popped into existence\n";
}

And it outputs: $foo{bar} popped into existence.

This is very confusing to me because I never did anything with $foo{bar} except check to see if it contained a reference to a hash (which I though was supposed to prevent autovivification).

My two questions are why does this happen (and why only with a hash of hashes) and how can I test for a hash within a hash without causing the key in question to come into existence?

Thanks!

[Aristotle]

Yes you did do something with it. In order to check whether exists $foo{bar}{baz}, obviously, $foo{bar} must be a hash reference. Perl DWIMs (most of the time) by creating an empty hashref on multilevel lookup attempts, but sometimes, like in your case, that is undesired. What you have to do is clunky: check every level of the hierarchy yourself. Something like

my $cursor = \%foo;
for(qw(foo bar)) {
    undef $cursor, last unless exists $cursor->{$_};
    $cursor = $cursor->{$_};
}

print "exists\n" if $cursor;

You want to put this in a function like

sub exists_novivify {
    my $cursor = shift;
    for(@_) {
        undef $cursor, last
            unless exists $cursor->{$_};
        $cursor = $cursor->{$_};
    }
    $cursor;
}

print "exists\n" if exists_novivify \%foo, qw(bar baz);

Update: removed duplicate exists. Thanks BrowserUk.

Makeshifts last the longest.

[Mr_Person]

Thanks! That sorta makes sense, although it's still annoying. I guess that's the price you pay for Perl's flexibility.

[BrowserUk]

exists tests for the existance of the last part of the expression. It will autovivify any earlier parts in the process. You need to code

<p><code>
if(exists($foo{bar}{baz})) {
  print "\$foo{bar}{baz} exists\n";
}

as

if(exists($foo{bar}) and exists($foo{bar}{baz})) {
  print "\$foo{bar}{baz} exists\n";
}

to prevent it.

---

Examine what is said, not who speaks.

The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

[Cabrion]

The use of eval might be in order here to programtically build up:

if(exists($foo{bar}) and exists($foo{bar}{baz})) {
  print "\$foo{bar}{baz} exists\n";
}

by doing soemthing like this:

my @levels = qw/bar baz/; 
foreach(@levels) {

$_ = 'exists('.$_.')';
}
my $code = '('.join(' and ', @levels).')';
#$code is now: (exists($foo{bar}) and exists($foo{bar}{baz})
if (eval $code) {
 #do whatever
}

Note that the order of @levels is important.

[BrowserUk]

No need to pay the penalty of eval, see Aristotle's exists_novivify sub

---

Examine what is said, not who speaks.

The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

[trs80]

An alternate way to test is by using the ref function, this would avoid autovivification in a way similar to the on the first key then the existence of key beneath it option suggested. That other method might not work correctly because 'bar' may be a real key, but its content might not be an anonymous hash. In other words the key exists so the first test would pass and then cause an error similar to:

Can't use string ("key bar's value") as a HASH ref while "strict refs"

The sub routine suggested by Aristotle suffers from the same problem. However there is no mention in the orignal post as to how complex the data structure might be, but I in my own experience I have been biten by similar problems.

Perl fails as early as possible in the truth process, that is the ref fails so it doesn't bother attempting to try the second operation, thereby avoiding the problem.
Here it is with the ref and no value for 'bar'

#!/usr/bin/perl

use strict;
use warnings;

my %foo;

if(exists($foo{qux})) {
    print "\$foo{qux} exists";
}

if(ref $foo{bar} eq 'HASH' && exists($foo{bar}{baz})) {
    print "\$foo{bar}{baz} exists\n";
}

if(exists($foo{bar})) {
    print "\$foo{bar} popped into existence\n";
}

Now if we assign a scalar value to bar and try again it still works.


#!/usr/bin/perl

use strict;
use warnings;

my %foo;

$foo{bar} = "key bar's value";

if(exists($foo{qux})) {
    print "\$foo{qux} exists";
}

if(ref $foo{bar} eq 'HASH' && exists($foo{bar}{baz})) {
    print "\$foo{bar}{baz} exists\n";
}

if(exists($foo{bar})) {
    print "\$foo{bar} popped into existence\n";
}

And finally with exist && exist to produce our error.

#!/usr/bin/perl

use strict;
use warnings;

my %foo;

$foo{bar} = "key bar's value";

if(exists($foo{qux})) {
    print "\$foo{qux} exists";
}

if(exists($foo{bar}) && exists($foo{bar}{baz})) {
    print "\$foo{bar}{baz} exists\n";
}

if(exists($foo{bar})) {
    print "\$foo{bar} popped into existence\n";
}

[pg]

I am quite confused with your second piece of code. What does it try to demo? What is your expected result (to print out the "popped into existence" or not, I tested it, and it printed)?

Also I don't know whether you realize that you actually used the concept of symblic reference in your second demo.

I changed your second demo a little bit, and added comments to explain:

use strict; #this basically disallows symblic ref
use warnings;

my %foo;

$foo{bar} = "key bar's value"; #now you made $foo{bar} EXISTS, there i
+s no need for auto-vivification to create it any more. MOST IMPORTANT
+LY, this can be a symblic reference depending on how you use it, cont
+inue...

if(exists($foo{qux})) {
    print "\$foo{qux} exists";
}

if(ref $foo{bar} eq 'HASH' && exists($foo{bar}{baz})) {#Here you are s
+trongly suggesting a symblic reference. As for the code, the part bef
+ore && obviously evaluate to false, so that exists after && will be e
+valuated, but no auto-vivfication here, as $foo{bar} exists any way
    print "\$foo{bar}{baz} exists\n";#will not print as there is no $f
+oo{bar}{baz}
}

if(exists($foo{bar})) {#yes
    print "\$foo{bar} popped into existence\n";#print out, but not pop
+ped into existence, it is actually created by you
}

#the following is added by me
$foo{bar}{baz} = 1; #violates strict refs, error, because you set $foo
+{bar} to a string earlier. There is no way to create $foo{bar}{baz} i
+n this context, unless you turn off use strict("refs")

update:

Yes, Aristotle is right, and I messed up with && and ||, Thanks for pointing out.

However this mistake does not reduce the value of this post, and the post as a whole is still correct, especially the discussion on symbolic reference.

I keep the mistake there (Remove the original mistake pointed out by other monks, would make other monk's reply funny, and sounds like irrelevant, and I don't do this ;-)



"And the latter is exactly what he's talking 
about protecting against by using ref."

Doubt.

[Aristotle]

Here you are strongly suggesting a symblic reference. As for the code, the part before && obviously evaluate to false, so that exists after && will be evaluated, but no auto-vivfication here, as $foo{bar} has been created by you earlier

Actually, a) the ref failing will immediately lead to the entire expression failing because he is using &&, not ||. b) If it got executed, strict would cause the exists to fail with a refs stricture violation.

And the latter is exactly what he's talking about protecting against by using ref.

Makeshifts last the longest.

[pg]

To auto-vivificate when you call exists or defined is conceptually wrong, and it is actually a reported bug. However it is not likely that this will be fixed before Perl 6. If you look into the c code, you would see how complex the fix would be.

In Perl 6, auto-vivification would still be there, and you know that it is extreamly helpful in creating complex reference structures.

But it would become more context aware, and most likely only auto-vivificate if you use the ref structure as lvalue.

[Pardus]

This is documented behaviour.

In most cases a "bug" like this is a sign the idea behind your hash tree is wrong. Use the hierarchy of the hash tree to your advantage or just make it a single hash.
--
Jaap Karssenberg || Pardus (Larus)? <[e-mail]>
>>>> Zoidberg: So many memories, so many strange fluids gushing out of patients' bodies.... <<<<

[ihb]

The why has been answered already, but only hand-rolled solutions has been showed, so I figured I should mention the module Hash::NoVivify, which provides two non-autovivifying functions: &Exists and &Defined.

ihb

[Anonymous Monk]

Here's a neat (I think so anyway) little trick that I found.
1) declare a hashref. ($hashref = {})
if you check to see if the hash is defined then
if(defined $hashref)
always returns true even if you have not added to the hash, however if you check for
if(defined %$hashref)
then this will only return true if $hashref actually refers to an existing hash. so I guess that
if(defined $hashref->{KEY})
should only return true if the key exists.

[Aristotle]

This is broken. It will falsely return false for keys which exist but point to a value equal to 0 or the empty string.

Nor does it solve the problem at hand, because exists $hashref->{KEY} will not autovivify that key into a hashref either. It is only when you try something like exists $hashref->{LEVEL1}->{LEVEL2} that the level one key is automagically created. But in that case, defined $hashref->{LEVEL1}->{LEVEL2} will do the same.

So your "solution" adds a new problem without solving the existing one.

Makeshifts last the longest.