Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re^3: Peculiar Reference To U+00FE In Text::CSV_XS Documentation

by Anonymous Monk
on Dec 10, 2012 at 05:42 UTC ( #1008043=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Peculiar Reference To U+00FE In Text::CSV_XS Documentation
in thread Peculiar Reference To U+00FE In Text::CSV_XS Documentation

I don't understand ... the documentation is ambiguous

But did you understand what I said? What number is it?

In the source

#define byte unsigned char typedef struct { byte quote_char; byte escape_char; byte sep_char;

255 is the biggest a byte gets, right?

:D


Comment on Re^3: Peculiar Reference To U+00FE In Text::CSV_XS Documentation
Download Code
Re^4: Peculiar Reference To U+00FE In Text::CSV_XS Documentation
by Jim (Curate) on Dec 10, 2012 at 06:20 UTC

    \xFE is 254, not 255. It's the second biggest a byte gets. \xFF is the biggest byte.

    In any case, the character with code point U+00FE isn't a single-byte character in any Unicode character encoding scheme.

    I suspect the peculiar reference to U+00FE in the documentation has something to do with the Concordance DAT file. I hope it does, because it would then imply that Text::CSV_XS can be used to parse Concordance DAT records, which is precisely what I need to do.

    Jim

      I do not recall exactly why the docs were written the way they were, but as I am unaware of the DAT format, I cannot verify that U+00FE was referred to because of this format.

      Knowing my own way of thinking, it most likely is not 0xFF, as that would be -1, which could be used as a guard marker or something alike (currently it isn't). I might have used 0xFE as it is the next highest byte.

      I have just read your post, and the only conflict I see that Text::CSV_XS is not able to do is optional line endings. The optional <CR> before the <NL> is automatically dealt with (just do not specify eol), but you cannot have an extra U+00AE to also end records. If otoh 0xAE is just a placeholder for embedded newlines, that is easy to do (see below).

      Another point of care is that Text::CSV_XS does not deal with BOM's, so you'll need File::BOM or other means to deal with that.

      my $csv = Text::CSV_XS->new ({ sep_char => "\x{14}", quote_char => "\x{fe}", escape_char => undef, binary => 1, auto_diag => 1, }); while (my $row = $csv->getline ($fh)) { tr/\x{ae}/\n/ for @$row; # continue as usual }

      If it doesn't, I'd like to see some data.

      Note that the encoded U+00FE is 0xC3BE, which is two bytes, and two bytes cannot be used as a sep_char in Text::CSV_XS, which parses the data as bytes, so the stream has to be properly coded before parsing.


      Enjoy, Have FUN! H.Merijn

        Thank you, Tux, for your reply.

        If otoh 0xAE is just a placeholder for embedded newlines, that is easy to do (see below).

        Yes, U+00AE (, REGISTERED SIGN, 0xC2 0xAE in UTF-8) is used as a placeholder for literal newlines in quoted strings. The CSV records in Concordance DAT files are ordinary ones with standard EOL characters:  <CR><LF> pairs.

        Another point of care is that Text::CSV_XS does not deal with BOM's, so you'll need File::BOM or other means to deal with that.

        This would be a nice feature to add to Text::CSV_XS:  proper handling of Unicode byte order marks in UTF-8, UTF-16 and UTF-32 CSV files.

        Note that the encoded U+00FE is 0xC3BE, which is two bytes, and two bytes cannot be used as a sep_char in Text::CSV_XS, which parses the data as bytes, so the stream has to be properly coded before parsing.

        This settles it. It's not the answer I'd hoped for, but I'm glad to know now with certainty that Text::CSV_XS cannot parse a UTF-8 Concordance DAT file. I'll stop trying hopelessly to make it work. ;-)

        How difficult would it be to enhance Text::CSV_XS to handle metacharacters in Unicode CSV files that are outside the Basic Latin block (i.e., not ASCII characters)? The Concordance DAT file is a de facto standard format for data interchange in the litigation support and e-discovery industry. As I've explained, the only thing special about it is the unusual and unfortunate characters it uses for metacharacters:  U+0014, which is a control code; U+00FE, which is word constituent character; and U+00AE, which is a common character in ordinary text.

        Jim

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1008043]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (6)
As of 2014-09-21 02:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (165 votes), past polls