Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re: CSV nightmare

by snoopy (Deacon)
on Jun 03, 2008 at 01:36 UTC ( #689784=note: print w/ replies, xml ) Need Help??

in reply to CSV nightmare

Also, there's another problem with your second solution. Unicode support remains on Text::CSV_XS's todo list.

You could instead try the largely compatible Text::CSV::Unicode.

#!/usr/bin/perl use warnings; use strict; # solution 2, converted to Text::CSV::Unicode # replacement for Text::CSV_XS use Text::CSV::Unicode; my $csv = Text::CSV::Unicode->new(); my $count=0; my $file = 'PS0002_9_2006b.txt'; if (defined $ARGV[0]) { $file = $ARGV[0]; } my $sum = 0; # ikegami's fix open(my $data, '<:raw:encoding(UCS-2le):crlf:utf8', $file) or die "Cou +ld not open '$file'\n"; while (my $line = <$data>) { chomp $line; exit if ($count >= 20); $count++; if ($csv->parse($line)) { my @columns = $csv->fields(); $sum += $columns[2]; } else { warn "Line could not be parsed: $line\n"; } } print "$sum\n"; # end solution 2_______________________

Comment on Re: CSV nightmare
Download Code
Re^2: CSV nightmare
by Tux (Monsignor) on Jun 03, 2008 at 08:50 UTC

    Text::CSV might soon be extended with a layer that deals with encodings, somewhat like this:

    use Text::CSV::Encoded; my $csv = Text::CSV::Encoded->new ({ encoding => "utf-8", # Both in and out encoding_in => "utf-16le", # Only the input encoding_out => "cp1252", # Only the output });

    Until then, I think

    binmode STDOUT, ":utf8"; my $csv = Text::CSV_XS->new ({ binary => 1 }); open my $fh, "<:encoding(utf-16le)", $file or die "$file: $!"; while (my $row = $csv->getline ($fh)) { print $row->[4]; }

    should work

    Enjoy, Have FUN! H.Merijn

      It's already been covered that it should be

      open my $fh, "<:raw:encoding(utf-16):crlf:utf8", $file or die "$file: $!";

      or more precisely,

      open my $fh, "<:raw:encoding(ucs-2le):crlf:utf8", $file or die "$file: $!"; read($fh, my $bom='', 1);

      And no, it doesn't work. Not if the data contains any non-ASCII characters, at least, but that's the whole point of this exercise. The UTF8 flag gets turned off, so the UTF-8 encoding of the characters is treated as iso-latin-1.

      For example, if a field contains <"é">, Text::CSV_XS returns the two characters <é> instead of <é>. (I'm using angled brackets to quote to avoid confusion with the double-quotes in the CSV file.)

      For example, if a field contains <"♠">, Text::CSV_XS returns the three characters <♣> instead of <♠>.

      The flag needs to be reinstated, so it should be:

      use Encode qw( _utf8_on ); my $csv = Text::CSV_XS->new ({ binary => 1 }); # UTF-16 or UCS-2 file with BOM and CRLF or LF line endings. open my $fh, "<:raw:encoding(utf-16):crlf:utf8", $file or die "$file: $!"; while (my $row = $csv->getline ($fh)) { # Fix inability of CSV_XS to handle UTF8 strings. _utf8_on($_) for @$row; print $row->[4]; }

      There is at least one other problem with treating characters encoded using UTF-8 no differently then characters encoded using iso-latin-1 as Text::CSV_XS does.

      If any of eol, sep_char, etc is passed a string with the UTF8 flag off and it contains a character in [\x80-\xFF], Text::CSV_XS can generate false positives. However, this is unlikely to ever happen.

      Text::CSV might soon be extended with a layer that deals with encodings

      I don't see the point, since Text::CSV doesn't open any file handles. All it needs to do is respect the UTF8 flag on strings it receives via getline, eol, sep_char, etc. Currently (well, 0.34 and presumably 0.45), it ignores it.

        Text::CSV_XS doesn't do anything with encodings internally, and reads bytes, which is also the reason why EBCDIC is so damn hard to implement in the current structure.

        That said, the user is resonsible for the encoding/decoding of the data/fields, as CSV files have no way of telling that to the parser.

        _utf8_on ($_) is NOT the way to go. Please read Unicode advice and Perl Unicode totorial for the reasons.

        while (my $row = $csv->getline ($fh)) { # Fix inability of CSV_XS to handle UTF8 strings. utf8::decode ($_) for @$row; print $row->[4]; }

        As a proof of concept, I tried something more simple in the example below

        #!/pro/bin/perl use strict; use warnings; use Text::CSV_XS; use Encode qw( encode decode ); my $csv = Text::CSV_XS->new ({ binary => 1 }); my $file = "test.csv"; open my $fh, ">:encoding(utf-16)", $file or die "$file: $!"; print $fh join ",", "\x{0073}e\x{00f1}\x{00f3}\x{0159}", 123, "\x{00c5}\x{0142}\x{00e9}\x{0161}\x{0171}\x{0146}", "\r\n"; close $fh; binmode STDOUT, ":utf8"; open $fh, "<:raw:encoding(utf-16)", $file or die "$file: $!"; while (my $row = $csv->getline ($fh)) { print join "," => @$row, "\n"; utf8::decode ($_) for @$row; print join "," => @$row, "\n"; }

        To show that test.csv now has a BOM:

        $ od -t x1 test.csv 0000000 fe ff 00 73 00 65 00 f1 00 f3 01 59 00 2c 00 31 0000020 00 32 00 33 00 2c 00 c5 01 42 00 e9 01 61 01 71 0000040 01 46 00 2c 00 0d 00 0a 0000050

        And the cript output was:


        The second line was exactly what I was expecting.

        Enjoy, Have FUN! H.Merijn

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://689784]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (8)
As of 2014-07-13 19:21 GMT
Find Nodes?
    Voting Booth?

    When choosing user names for websites, I prefer to use:

    Results (251 votes), past polls