I have modified your code to process the incoming stream looking for min, max and unique values. Configure @min_, @max_ and @unique_ columns to control which fields are processed.
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV_XS;
use Data::Dumper;
my $file = "./test.csv";
my @min_columns = qw( ozone ozone_f qa_code ozone_8hr);
my @max_columns = qw( ozone ozone_f qa_code rank );
my @unique_columns = qw( site_id );
my $data;
my $counter;
my $csv = Text::CSV_XS->new({
binary => 1,
auto_diag => 1,
}) or die "Cannot use CSV: " . Text::CSV_XS->error_diag();
open my $FH2, "<:encoding(utf8)", $file or die "$file: $!";
my $columns = $csv->getline($FH2);
for (@$columns) {s/^\s+//; s/\s+$//};
$csv->column_names(@$columns);
while (my $row = $csv->getline_hr($FH2)) {
process_row($row);
}
print "done with loop after $counter rows\n";
$csv->eof or $csv->error_diag();
close $FH2;
print Dumper $data;
sub process_row
{
my $row = shift;
$counter++;
print "processing: $counter\n" if ($counter % 100 == 0);
for my $column (@min_columns) {
my $val = $row->{$column};
my $d = $data->{$column}{min};
my $update = {value => $val, row => $row};
$data->{$column}{min} =
!exists $d->{value} ? $update
: $d->{value} > $val ? $update
: $d;
}
for my $column (@max_columns) {
my $val = $row->{$column};
my $d = $data->{$column}{max};
my $update = {value => $val, row => $row};
$data->{$column}{max} =
!exists $d->{value} ? $update
: $d->{value} > $val ? $update
: $d;
}
for my $column (@unique_columns) {
my $val = $row->{$column};
$data->{$column}{unique}{$val}++;
#could just be =1, but now we can get unique and count distinc
+t
}
}
I switched to Text::CSV_XS from _PP, as the example data had an issue around line 296. (PP parse error 2025, loose escape char)
This is an excellent spot to use DataCube, but that module is gone now. I would like David to fix it and put it back on CPAN. backpan DataCube
If you're going to be doing ad-hoc analysis of csv files, using r language may prove useful.
Edit: added updates from Tux's comment