Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re: Regexp and Linux (is it utf issue?)

by aitap (Curate)
on Jun 28, 2013 at 06:21 UTC ( #1041153=note: print w/replies, xml ) Need Help??

in reply to Regexp and Linux (is it utf issue?)

When you read text files, you should decode them. This is easy using PerlIO layers, Encode module and three-argument form of open:

use Encode; open my $fh, "<:encoding(whatever)", $filename or die $!;
This way, Perl decodes everything automatically, and you only have to work with characters, not bytes.

When you write text to files, writing characters produces the famous warning: "wide character in (sub name)...". You need to encode them using the same technique: open my $write, ">:encoding(whatever)", $filename or die $!;. You can use :utf8 layer to encode characters because they are internally stored as valid UTF-8.

Do not use :utf8 iolayer to decode text because it simply sets "character" flag on the strings read from filehandles without any checks and this is generally unsafe: UTF8 related proof of concept exploit released at T-DOSE.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1041153]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (8)
As of 2018-05-28 10:11 GMT
Find Nodes?
    Voting Booth?