Re: Data munging - in, out and something in between

[Anonymous Monk]

Sorry for the lack of code in my original question; it is not so easy to demonstrate in a code snippet, and impossible to show the data, but here goes...

    open(IN,  " < $infile" );

    my $si=8192;  # the number of entries is known

    my %matrix;

    while(my $c<$si) {

        read(IN, my $bin, 4);
        $data = unpack('N', $bin);          
            
        my $d = Unshuffle($c);
          
        $matrix{$d} += $data;
            
        $c++;
    }

    close( IN );

    open(OUT, " >$outfile" );

    foreach my $d (sort {$a <=> $b} (keys %matrix)) {
        print (OUT pack('V', $matrix{$d}));
    }

    close( OUT );

The subroutine Unshuffle() simply calculates a new position in the output matrix for each entry. The above works until I try to scale the data, e.g.  $matrix{$d} += $data/3; In which case the output is scrambled. BTW, null opperations like:  $matrix{$d} += $data/1;  $matrix{$d} += $data+0; still work. This, I suspect, explains the fact that the script also fails if more than one of data points added to make $matrix{$d} is non-zero. I guess that my number is now too long for the "V" template in pack. Yes, the template must be 'V', otherwise a downstream program spits the dummy. thanks again

[rodion]

I tried your code on my machine, with a few additions/modifications to get it running, all of them marked with "# **" below. I didn't see any problem with the output. The only really significant modifications were to take the "my" away from the "$c" in the while loop, since it was re-initializing $c with each pass through the loop, and to add binmode(IN), which actually doesn't make any difference with the data I'm using.

Try this modified code on your machine and see if you get appropriate output. It only processes the first two numbers, so it should be easy to see what's going on and play with it.

use warnings;                   # **
use strict;                     # **

my $data;                       # **
my $c = 0;                      # **
my $outfile = 'tempo.pk';       # **
my $infile = shift;             # **

open(IN,  " < $infile" );
binmode(IN);                    # ** prevents newline translation
my $si=2;   # the number of entries is known
my %matrix;
while($c<$si) {                 # ** removed my
    read(IN, my $bin, 4);
    $data = unpack('N', $bin);  

    my $d = Unshuffle($c);

    print "$c -> $d ($data)\n";  # **
    $matrix{$d} += $data/3;      # **

    $c++;
}
close( IN );
open(OUT, " >$outfile" );
foreach my $d (sort {$a <=> $b} (keys %matrix)) {
    print "$outfile $d ($matrix{$d})\n"; # **
    print (OUT pack('V', $matrix{$d}));
}
close( OUT );

open(IN, " <$outfile" );         # **
while( read(IN, my $bin, 4) ) {  # **
   $data = unpack('V', $bin);    # **
   print "$data:";               # **
}                                # **


sub Unshuffle { # ** interchange the first two positions
   my $val = shift;
   return 1   if ($val == 0);
   return 0   if ($val == 1);
   return $val;
}
# output was
#  0 -> 1 (2140)
#  1 -> 0 (1544028160)
#  tempo.pk 0 (514676053.333333)
#  tempo.pk 1 (713.333333333333)
#  514676053:713: