Re^3: Processing ~1 Trillion records

[Mr. Muskrat]

in reply to Re^2: Processing ~1 Trillion records
in thread Processing ~1 Trillion records

You want to read up on bind_columns because it "is the most efficient way to fetch data" (according to the DBI docs).

my $sth_chr = $dbh->prepare( $sql_chr );
$sth->execute;
my %row;
my $rv = $sth_chr->bind_columns( \( @row{ @{$sth->{NAME_lc} } } ) ); #
+ from the DBI docs
while ($sth->fetch) {
    # access data with $row{column_a}, $row{column_b}, etc...
}

If the real query selects columns that are restricted to one value in the where clause then you can speed it up tremendously by not returning the static data. In the example query, you only care about column_c so you could remove everything else from the select (including the join). If this isn't the case, you might want to come up with a better example.