Re: Text::CSV and Unicode

in reply to Text::CSV and Unicode

This has come up before, and as of Text::CSV_XS version 1.00, the behavior is now consistent. It however does not meet your current needs. I just uploaded version 1.02 a minute ago, as that now has a new attribute decode_utf8 that enables you to disable the default behavior (which is what has proven to be what most people want and expect).

       decode_utf8
           This attributes defaults to TRUE.

           While parsing, fields that are valid UTF-8, are automatical
+ly set
           to be UTF-8, so that

             $csv->parse ("\xC4\xA8\n");

           results in

             PV("\304\250"\0) [UTF8 "\x{128}"]

           Sometimes it might not be a desired action. To prevent thos
+e
           upgrades, set this attribute to false, and the result will 
+be

             PV("\304\250"\0)
[download]

I realize that "most people" is not "all people" and I cannot make a default that makes "all" people happy. That is also the reason why I waited with 1.02. I have asked many users about what should be the default and also check the historical entries in RT and my mail and came to the conclusion that nowadays the majority works with UTF8 CSV more than with binary CSV. The change in 1.00 was not to enable UTF-8 or to disable it. The change was to make it work more consistently.

Enjoy, Have FUN! H.Merijn

Comment on Re: Text::CSV and Unicode Select or Download Code

Replies are listed 'Best First'.
Re^2: Text::CSV and Unicode by vsespb (Chaplain) on Nov 11, 2013 at 11:00 UTC
Ok, thank you ! I will try new version conclusion that nowadays the majority works with UTF8 CSV That's no problem, I just wanted this behaviour to be clearly documented (I did not want to rely on something undocumented).	[reply]
Re^2: Text::CSV and Unicode by Jim (Curate) on Nov 11, 2013 at 17:59 UTC
Tux, Hello! And thank you for doing such a fine job of maintaining Text::CSV_XS. Does the release of Text::CSV_XS version 1.02 mean that I can now use Text::CSV_XS to parse UTF-8 Concordance DAT files, something it couldn't do before? The peculiar characteristics of a Concordance DAT file are that it uses a control character as the separator character (U+0014) and a non-ASCII/non-ISO-8859-1 character as the string delimiter character (U+00FE), which is encoded with two bytes in UTF-8. (See Best Way To Parse Concordance DAT File Using Modern Perl? and Peculiar Reference To U+00FE In Text::CSV_XS Documentation.) Jim	[reply]
Re^3: Text::CSV and Unicode by Anonymous Monk on Nov 11, 2015 at 15:09 UTC
Oddly my similar issues with Text::CSV were resolved when I simply installed Text::CSV_XS. There must be a shared library that gets updated or something...	[reply]

In Section Seekers of Perl Wisdom