Re^2: Hashes of Hashes of Arrays? Is it possible?

[capriguy84]

Thanks for your reply. I am reading a CSV file(below) and creating a temporary complex hash structure as described to perform some operations. As you can see my $key is the time which should be only one per second and plug the data as hash/array.

NAME,06/01/2011,09:30:00,16.76,756,Q,00
NAME,06/01/2011,09:30:00,16.76,300,Q,00
NAME,06/01/2011,09:30:00,16.76,100,Q,00
NAME,06/01/2011,09:30:01,16.76,100,Q,00
NAME,06/01/2011,09:30:01,16.76,200,Z,00
NAME,06/01/2011,09:30:02,16.77,200,X,00
NAME,06/01/2011,09:30:02,16.77,200,X,00

Open 100k line CSV file and read using Text_CSV. I have to undef the hashes before jumping to next second.

    my $csv = Text::CSV_XS->new();
    open (FILE, "<", "$tickdir/$file") or die "Can't open CSV File:$! 
+\n";

    while (<FILE>) {
        $csv->parse($_);
        my @columns = $csv->fields();
        my $key = str2time($columns[1].' '.$columns[2]);
        my (@prices, @volumes) = ();
        my %HOH = ();

        $HOH{$key}{"name"} = $columns[0];
        push(@{$HOH{$key}{"price"}}, $columns[3]);
        push(@{$HOH{$key}{"volumes"}}, $columns[4]);

        print Dumper(\%HOH);
    }
    close FILE;

OUTPUT: As you can see the key values are same.

$VAR1 = {
          '1306935005' => {
                            'volumes' => [
                                           '200'
                                         ],
                            'name' => 'NAME',
                            'price' => [
                                         '16.76'
                                       ]
                          }
        };
$VAR1 = {
          '1306935005' => {
                            'volumes' => [
                                           '400'
                                         ],
                            'name' => 'NAME',
                            'price' => [
                                         '16.76'
                                       ]
                          }
        };

[toolic]

You are overwriting your hash every time through the while loop. Move the declaration of the hash outside of the loop:

    my $csv = Text::CSV_XS->new();
    open (FILE, "<", "$tickdir/$file") or die "Can't open CSV File:$! 
+\n";

    my %HOH;
    while (<FILE>) {
        $csv->parse($_);
        my @columns = $csv->fields();
        my $key = str2time($columns[1].' '.$columns[2]);
        my (@prices, @volumes) = ();

        $HOH{$key}{"name"} = $columns[0];
        push(@{$HOH{$key}{"price"}}, $columns[3]);
        push(@{$HOH{$key}{"volumes"}}, $columns[4]);

        print Dumper(\%HOH);
    }
    close FILE;

[capriguy84]

ahh..That fixed it. Thank you very much. One quick question, how do I avoid having to save the huge dataset in memory by processing and printing the results on fly? I would like to undef the HASH at after each second in time i.e create a loop for each key and undef my %HOH. Any ideas how/where I can add the control statement for $key?

[Marshall]

So as I understand it, you want to dump the current results and start a new %HOH when the $key changes.

When setting something like this up there are 3 things to consider:
1. What is the steady state, normal thing causes the desired action?
2. How do we get it started? (what is special about it?)
3. How do we get it stopped? (what is special about it?)

One approach for is to dump the results and reset %HOH when the $key changes. To do that perhaps:

my $prev_key = "";
my %HOH;
while ( a line read from file )
{
   ...get the current key .. code that you have already
   if (  $current_key ne $prev_key)
   {
      if ($prev_key ne "" ) #don't dump first time through!
      {
         ...dump the %HOH  #this is the previous result
      }
      $prev_key = $current_key;
      ...%HOH=();  #reset!
   }
   
   ...processing of the current_key
}
...dump the %HOH ... #the last record...

To solve question (2), the first time through the loop, I don't dump the %HoH unless there is data there - there is more than one possible way to test for this (keys %HOH) or ($prev_key ne "" ). If the thing that does the dump/final processing of this time slice has some code to not do anything if there is no data, there is no nested "if".

To solve question (3), after the loop ends due to lack of data, dump the final incarnation of the hash.

There is of course more than one way to accomplish what you need, but the 3 questions are always the same : (1)steady-state operation, (2)getting started, (3)getting finished.

I recommend writing the "steady-state", question (1) code first and then "fiddle with it" until questions (2) and (3) work out ok.

[Jim]

One quick question, how do I avoid having to save the huge dataset in memory by processing and printing the results on fly? I would like to undef the HASH at after each second in time i.e create a loop for each key and undef my %HOH.

For that, you want to introduce a simple control break into your program.

[capriguy84]

Jim, Is there a Perl specific control break? Or should I just permutate various loop control statements like unless, continue etc.?

[Jim]

Is there a Perl specific control break?

No. A control break is a general structured programming technique. It's not a feature of a specific programming language.

Or should I just permutate various loop control statements like unless, continue etc.?

Control breaks are implemented using if-then(-else) constructs inside and after an ordered-record-processing loop. Marshall explains the control break technique nicely in his reply to your inquiry.