Re^3: Huge data file and looping best practices

[BrowserUk]

Would it be worthwhile to pre-allocate the @patNos and @data arrays with $#array = N_000_000; statements?

It wouldn't hurt, but would make little difference to the overall runtime given the scale of the processing.

Even though it would lead to 8,000,000 redundant entries giving the variance of each patent with itself (i.e., 0), might it be profitable to eliminate the next if $first == $second; statement from the inner loop of the variance calculation section? Or, avoiding redundant entries, to break the inner loop into two consecutive loops with identical BLOCK contents: for my $second (0 .. $first-1) { ... } and then for my $second ($first+1 .. $#patNos) { ... } ?

With time to give it a little more thought, there is no point in comparing both patent 1 with patent 2 & patent 2 with patent 1. With that insight, the inner loop should run from $first +1 .. $#patNos - 1. That reduces the variance calculations from 64e12 to 32e12 - 8e6 for a further halving of the runtime:

#! perl -sw
use 5.010;
use strict;

my( $i, @patNos, @data ) = 0;
$#patNos = $#data = 8e6;

while( <> ) {
    my @bits =split ',';
    $patNos[ $i ] = shift @bits;
    $data[ $i++ ] = pack 'b480', @bits;
    print "\r$.\t" unless $. % 1000;
}
say "\n", time;

open OUT, '>', 'variances.csv' or die $!;
my @variances;
for my $first ( 0 .. $#patNos-1 ) {
    for my $second ( $first+1 .. $#patNos ) {
         say OUT "$patNos[ $first ], $patNos[ $second ], ",
            unpack '%32b*', ( $data[ $first ] ^ $data[ $second ] );
    }
}
close OUT;