Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS

[Jim]

Here's an excerpt from the documentation of Text::CSV_XS:

The separation-, escape- [sic], and escape- characters can be any ASCII character in the range from 0x20 (space) to 0x7E (tilde). Characters outside this range may or may not work as expected. Multibyte characters, like U+060c (ARABIC COMMA), U+FF0C (FULLWIDTH COMMA), U+241B (SYMBOL FOR ESCAPE), U+2424 (SYMBOL FOR NEWLINE), U+FF02 (FULLWIDTH QUOTATION MARK), and U+201C (LEFT DOUBLE QUOTATION MARK) (to give some examples of what might look promising) are therefor [sic] not allowed.

If you use perl-5.8.2 or higher, these three attributes are utf8-decoded, to increase the likelihood of success. This way U+00FE will be allowed as a quote character.

Is it possible to use \x{fe} for the quote_char character? If not, why not? Why is the Unicode code point U+00FE specifically mentioned in the documentation?

Here's a script I used to test using \x{14} for the sep_char character and \x{fe} for the quote_char character:

#!perl

use strict;
use warnings;

use Text::CSV_XS;

my $dat_file = <<EOF;
\x{feff}\x{fe}Row1Col1\x{fe}\x{14}\x{fe}Row1Col2\x{fe}\x{14}\x{fe}Row1
+Col3\x{fe}
Row2Col1\x{14}Row2Col2\x{14}Row2Col3
Row3Col1\x{14}\x{fe}Row3Col2\x{fe}\x{14}\x{fe}R\x{14}ow3Col3\x{fe}
\x{fe}Row4Col1\x{fe}\x{14}Row4Col2\x{14}\x{fe}Row4C\x{fe}\x{fe}ol3\x{f
+e}
Row5Col1\x{14}Row5Col2\x{14}\x{fe}Row5\nCol3\x{fe}
EOF

my $dat = Text::CSV_XS->new({
    sep_char       => "\x{14}",
    quote_char     => "\x{fe}",
    escape_char    => "\x{fe}",
    eol            => $/,
    binary         => 1,
    keep_meta_info => 1,
    auto_diag      => 1,
});

open my $dat_fh, '<:encoding(UTF-8)', \$dat_file;

binmode STDOUT, ':encoding(UTF-8)';

while (my $row = $dat->getline($dat_fh)) {
    my $r = $.;

    for my $c (1 .. @$row) {
        my $is_quoted = $dat->is_quoted($c - 1)
                      ? 'Is Quoted'
                      : 'Is Not Quoted';

        print "Row $r, Col $c $is_quoted:  $row->[$c - 1]\n";
    }
}

$dat->eof() or $dat->error_diag();

close $dat_fh;

exit 0;

Here's the output of the script:

Row 1, Col 1 Is Not Quoted:  þRow1Col1þ
Row 1, Col 2 Is Not Quoted:  þRow1Col2þ
Row 1, Col 3 Is Not Quoted:  þRow1Col3þ
Row 2, Col 1 Is Not Quoted:  Row2Col1
Row 2, Col 2 Is Not Quoted:  Row2Col2
Row 2, Col 3 Is Not Quoted:  Row2Col3
Row 3, Col 1 Is Not Quoted:  Row3Col1
Row 3, Col 2 Is Not Quoted:  þRow3Col2þ
Row 3, Col 3 Is Not Quoted:  þR
Row 3, Col 4 Is Not Quoted:  ow3Col3þ
Row 4, Col 1 Is Not Quoted:  þRow4Col1þ
Row 4, Col 2 Is Not Quoted:  Row4Col2
Row 4, Col 3 Is Not Quoted:  þRow4Cþþol3þ
Row 5, Col 1 Is Not Quoted:  Row5Col1
Row 5, Col 2 Is Not Quoted:  Row5Col2
Row 5, Col 3 Is Not Quoted:  þRow5
Row 6, Col 1 Is Not Quoted:  Col3þ

Obviously, the \x{fe} characters aren't being recognized as quote and escape characters.

(I'm running Perl 5.14.2 and Text::CSV_XS 0.90.)

Jim