Re: Text CSV_XS memory crash

[spazm]

I have modified your code to process the incoming stream looking for min, max and unique values. Configure @min_, @max_ and @unique_ columns to control which fields are processed.

#!/usr/bin/perl 
use strict;
use warnings;

use Text::CSV_XS;
use Data::Dumper;

my $file           = "./test.csv";
my @min_columns    = qw( ozone ozone_f qa_code ozone_8hr);
my @max_columns    = qw( ozone ozone_f qa_code rank );
my @unique_columns = qw( site_id );

my $data;
my $counter;

my $csv = Text::CSV_XS->new({ 
  binary    => 1,
  auto_diag => 1,
}) or die "Cannot use CSV: " . Text::CSV_XS->error_diag();

open my $FH2, "<:encoding(utf8)", $file or die "$file: $!";

my $columns = $csv->getline($FH2);
for (@$columns) {s/^\s+//; s/\s+$//};
$csv->column_names(@$columns);

while (my $row = $csv->getline_hr($FH2)) {
    process_row($row);
}
print "done with loop after $counter rows\n";
$csv->eof or $csv->error_diag();
close $FH2;

print Dumper $data;

sub process_row
{
    my $row = shift;
    $counter++;
    print "processing: $counter\n" if ($counter % 100 == 0);

    for my $column (@min_columns) {
        my $val    = $row->{$column};
        my $d      = $data->{$column}{min};
        my $update = {value => $val, row => $row};

        $data->{$column}{min} =
            !exists $d->{value} ? $update
          : $d->{value} > $val  ? $update
          :                       $d;
    }
    for my $column (@max_columns) {
        my $val    = $row->{$column};
        my $d      = $data->{$column}{max};
        my $update = {value => $val, row => $row};

        $data->{$column}{max} =
            !exists $d->{value} ? $update
          : $d->{value} > $val  ? $update
          :                       $d;
    }
     for my $column (@unique_columns) {
        my $val = $row->{$column};
        $data->{$column}{unique}{$val}++;
        #could just be =1, but now we can get unique and count distinc
+t
    }
}

I switched to Text::CSV_XS from _PP, as the example data had an issue around line 296. (PP parse error 2025, loose escape char)

This is an excellent spot to use DataCube, but that module is gone now. I would like David to fix it and put it back on CPAN. backpan DataCube

If you're going to be doing ad-hoc analysis of csv files, using r language may prove useful.

Edit: added updates from Tux's comment