Looping through a hash reference is creating a key...?

[the_slycer]

I could swear I've seen this behavior before, but can't find the post.

#!/usr/bin/perl
use warnings;
use strict;
use Data::Dumper;

my $questions = {
    1 => { foo => 'bar' },
    2 => { foo => 'bar' },
    3 => { foo => 'bar' }
};

print "(before):\n",Dumper($questions),"\n";

for (1 .. 4) {
    my $foo = $questions->{$_}{foo} || "daklfjsdalk";
}

print "(after): \n",Dumper($questions), "\n";

This returns something like:

(before):
$VAR1 = {
          1 => {
                 'foo' => 'bar'
               },
          2 => {
                 'foo' => 'bar'
               },
          3 => {
                 'foo' => 'bar'
               }
        };

(after): 
$VAR1 = {
          1 => {
                 'foo' => 'bar'
               },
          2 => {
                 'foo' => 'bar'
               },
          3 => {
                 'foo' => 'bar'
               },
          4 => {}
        };

And I'm not sure why, my question is twofold:
1) Is there a good reason why perl creates the reference (4)?
2) Is there a good reason why perl doesn't WARN that I am trying to access a value that doesn't exist? Note that even taking out the || "djkafdkljfa" does not produce a warning

The problem is appearing in a bit of code I am developing, and I can get around it, but this tripped me up while I was working on it (using exists($hashref->{key} to determine execution path).

ps:
Damn, I miss this place, I think it's time to start hanging out here again (especially now that I "do" perl for a living :))

[Roy Johnson]

Autovivification. Because you have to dereference $questions{4} to get to $questions{4}{foo}, the reference $questions{4} must be created.

Update: Note that if the not-already-existing reference is not dereferenced, but is just read, no autovivification goes on, so your code would work as you'd like if you did:

 for (1 .. 4) {
    my $foo = $questions->{$_} && $questions->{$_}{foo} || "daklfjsdal
+k";
}

[perrin]

Autovivification is the reason. If you want to avoid this, either use exists() before reading values or look at locked hashes using Hash::Util.

[Roy Johnson]

I'm going to quibble with your advice regarding exists, which could be right or wrong depending on how one reads it. (I don't doubt that your understanding is correct, but I think others might read it differently than you intended.) Autovivification (the creation of intermediate references) is not prevented by using exists, so exists $h{1}{2} still creates $h{1}.

To prevent autovivification, one would have to test exists $h{1} and only if it were true, examine $h{1}{2}. However, they could just test whether $h{1} were true and get pretty much the same result -- if it didn't exist before, it won't exist after. The only difference in using exists is if $h{1} has a non-reference value like zero, empty string, or undef. It seems a little odd to dereference it in those cases. Perhaps a better idea would be to test it with ref.

[perrin]

Good point. I remember finding that out the hard way when I wrote some code that did exists() checks on deeply nested nodes in a huge data structure. My exists checks made the structure grow about 10MB in memory! Adding uglier but more careful code (if (exists $h{1} && exists $h{1}->{2} && exists ...) made the problem go away.

[drewbie]

FYI, the hash locking feature only started in 5.8.0. From the docs.

Restricted hashes

5.8.0 introduces the ability to restrict a hash to a certain set of keys. No keys outside of this set can be added. It also introduces the ability to lock an individual key so it cannot be deleted and the value cannot be changed.

This is intended to largely replace the deprecated pseudo-hashes.

And hooray for removing pseudohashes! I have the notes from Damian Conway's Advanced OO Perl talk (Boston 2001) that talks a bit about pseudohashes. They were a cool idea, but the implementation was very flawed. The more you read about them, the less interested you became.

[Anonymous Monk]

Perl does something called autovivication. How you get around it is exactly what you've done here using the 'exists' test. Autovivication can be your friend at times, but it can really bite you too.

[imcsk8]

i know that this is not an answer regarding autovivification, but i think this comment establishes a point.
i think the problem is the approach you are choosing, if you are using numbers as hash keys, you should use an array (or an anonymous array reference) to store the hashes:
(untested)

my @questions = (
    { foo => 'bar' },
    { foo => 'bar' },
    { foo => 'bar' }
);

and get the values using a more natural loop:


foreach $hashref (@questions){
      my $foo = $hashref->{foo} || "daklfjsdalk";
}


or


for(my $i = 0;$i<=$#questions;$i++){
       my $foo = $questions[$i]->{foo} || "daklfjsdalk";
}


i know that this does not solve the issue, but sometimes different approaches can eliminate ambiguities on the code and the data results

i hope this would be useful

[sacked]

Also see Uri Guttman's article on autovivification. It includes some sample code to check for hash keys at any level of a multi-level data structure without triggering autovivification. (paraphrased from description in article)

http://tlc.perlarchive.com/articles/perl/ug0002.shtml

[the_slycer]

Thank you all for the responses, just to address a few points brought up..

First and formost, the code above is not representative of the data structure I was using other than in a vague, hash of hashrefs type of way. I do use numbers, but theyre not sorted etc, still need to be in a hash :)

Secondly, with regards to "exists", the problem became apparent because I was attempting to use exists as part of the flow of the code.. briefly:

$asked = exists($href->{$qid});

if ($asked) { #do stuff }
else { #do other stuff }

# later

%data_structure = (
  key => value,
  key2 => value2,
  key3 => $href->{$qid}{answer} || 'stuff',
  #more assignments
);

#do things with the data structure

The first loop through the code would work fine, further loops would later would raise warnings (when doing data validation). Once I saw/replicated the problem, it was the lack of warnings on the first loop that raised my eyebrows.

I hadn't heard of locked hashes, time to do some reading (though they don't look applicable for this particular issue at first glance)