Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS

by Jim (Curate)
on Jul 09, 2012 at 23:27 UTC ( #980781=perlquestion: print w/ replies, xml ) Need Help??
Jim has asked for the wisdom of the Perl Monks concerning the following question:

Here's an excerpt from the documentation of Text::CSV_XS:

The separation-, escape- [sic], and escape- characters can be any ASCII character in the range from 0x20 (space) to 0x7E (tilde). Characters outside this range may or may not work as expected. Multibyte characters, like U+060c (ARABIC COMMA), U+FF0C (FULLWIDTH COMMA), U+241B (SYMBOL FOR ESCAPE), U+2424 (SYMBOL FOR NEWLINE), U+FF02 (FULLWIDTH QUOTATION MARK), and U+201C (LEFT DOUBLE QUOTATION MARK) (to give some examples of what might look promising) are therefor [sic] not allowed.
If you use perl-5.8.2 or higher, these three attributes are utf8-decoded, to increase the likelihood of success. This way U+00FE will be allowed as a quote character.

Is it possible to use \x{fe} for the quote_char character? If not, why not? Why is the Unicode code point U+00FE specifically mentioned in the documentation?

Here's a script I used to test using \x{14} for the sep_char character and \x{fe} for the quote_char character:

#!perl use strict; use warnings; use Text::CSV_XS; my $dat_file = <<EOF; \x{feff}\x{fe}Row1Col1\x{fe}\x{14}\x{fe}Row1Col2\x{fe}\x{14}\x{fe}Row1 +Col3\x{fe} Row2Col1\x{14}Row2Col2\x{14}Row2Col3 Row3Col1\x{14}\x{fe}Row3Col2\x{fe}\x{14}\x{fe}R\x{14}ow3Col3\x{fe} \x{fe}Row4Col1\x{fe}\x{14}Row4Col2\x{14}\x{fe}Row4C\x{fe}\x{fe}ol3\x{f +e} Row5Col1\x{14}Row5Col2\x{14}\x{fe}Row5\nCol3\x{fe} EOF my $dat = Text::CSV_XS->new({ sep_char => "\x{14}", quote_char => "\x{fe}", escape_char => "\x{fe}", eol => $/, binary => 1, keep_meta_info => 1, auto_diag => 1, }); open my $dat_fh, '<:encoding(UTF-8)', \$dat_file; binmode STDOUT, ':encoding(UTF-8)'; while (my $row = $dat->getline($dat_fh)) { my $r = $.; for my $c (1 .. @$row) { my $is_quoted = $dat->is_quoted($c - 1) ? 'Is Quoted' : 'Is Not Quoted'; print "Row $r, Col $c $is_quoted: $row->[$c - 1]\n"; } } $dat->eof() or $dat->error_diag(); close $dat_fh; exit 0;

Here's the output of the script:

Row 1, Col 1 Is Not Quoted:  ■Row1Col1■
Row 1, Col 2 Is Not Quoted:  ■Row1Col2■
Row 1, Col 3 Is Not Quoted:  ■Row1Col3■
Row 2, Col 1 Is Not Quoted:  Row2Col1
Row 2, Col 2 Is Not Quoted:  Row2Col2
Row 2, Col 3 Is Not Quoted:  Row2Col3
Row 3, Col 1 Is Not Quoted:  Row3Col1
Row 3, Col 2 Is Not Quoted:  ■Row3Col2■
Row 3, Col 3 Is Not Quoted:  ■R
Row 3, Col 4 Is Not Quoted:  ow3Col3■
Row 4, Col 1 Is Not Quoted:  ■Row4Col1■
Row 4, Col 2 Is Not Quoted:  Row4Col2
Row 4, Col 3 Is Not Quoted:  ■Row4C■■ol3■
Row 5, Col 1 Is Not Quoted:  Row5Col1
Row 5, Col 2 Is Not Quoted:  Row5Col2
Row 5, Col 3 Is Not Quoted:  ■Row5
Row 6, Col 1 Is Not Quoted:  Col3■

Obviously, the \x{fe} characters aren't being recognized as quote and escape characters.

(I'm running Perl 5.14.2 and Text::CSV_XS 0.90.)

Jim

Comment on Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS
Select or Download Code
Re: Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS
by frozenwithjoy (Curate) on Jul 10, 2012 at 00:02 UTC

    To get things started, I was having major issues until I changed your two instances of encoding(UTF-8) to utf8.

    Next, are you sure you want \x{feff}? When I get rid of it, I start getting things quoted:

    Row 1, Col 1 Is Quoted: Row1Col1 Row 1, Col 2 Is Quoted: Row1Col2 Row 1, Col 3 Is Quoted: Row1Col3 Row 2, Col 1 Is Not Quoted: Row2Col1 Row 2, Col 2 Is Not Quoted: Row2Col2 Row 2, Col 3 Is Not Quoted: Row2Col3 Row 3, Col 1 Is Not Quoted: Row3Col1 Row 3, Col 2 Is Quoted: Row3Col2 Row 3, Col 3 Is Quoted: Row3Col3 Row 4, Col 1 Is Quoted: Row4Col1 Row 4, Col 2 Is Not Quoted: Row4Col2 Row 4, Col 3 Is Quoted: Row4C■ol3 Row 6, Col 1 Is Not Quoted: Row5Col1 Row 6, Col 2 Is Not Quoted: Row5Col2 Row 6, Col 3 Is Quoted: Row5 Col3

    also, it gets rid of the weird characters. One more thing, your input file isn't formatted consistently... There are some /n and double \x{fe}\x{fe} characters where I don't think you intended them to be.

      Thank you, frozenwithjoy, for your reply. I appreciate it.

      To get things started, I was having major issues until I changed your two instances of encoding(UTF-8) to utf8.

      My use of :encoding(UTF-8) instead of :encoding(utf8) is intentional and purposeful.

      Next, are you sure you want \x{feff}?

      I'm absolutely sure. It's the Unicode byte order mark, and it's vital to my test because the real CSV files I must process invariably have the byte order mark in them, encoded in UTF-8 (EF BB BF).

      When I get rid of it, I start getting things quoted: … also, it gets rid of the weird ■ characters.

      This doesn't happen for me. I get the same unhelpful results regardless of the presence or absence of the Unicode byte order mark. What versions of Perl and Text::CSV_XS are you using?

      One more thing, your input file isn't formatted consistently... There are some /n and double \x{fe}\x{fe} characters where I don't think you intended them to be.

      Those literal newlines and Latin small thorns (þ) are intentional and purposeful. They're vital to my test.

      Jim

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://980781]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (9)
As of 2014-12-25 22:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (163 votes), past polls