Re: UTF-8 and XML::Parser

[remiah]

in reply to UTF-8 and XML::Parser

Hello.

You need "use utf8;" for literal strings to be treated as utf8 "character".

#!/usr/bin/perl

use XML::Parser;
use utf8;           ###added
$xml = "<word>Müller</word>";

$ch = sub {
my ($p, $w) = @_;
    binmode STDOUT, ":encoding(UTF-8)"; ###added
    print "#$w#\n";
};

# the next commented 2 lines mean the same
# and translate the output to iso-8859-1
# $p = XML::Parser->new();
$p = XML::Parser->new(ProtocolEncoding => 'UTF-8');

# this line do the right job, but why??
#$p = XML::Parser->new(ProtocolEncoding => 'ISO-8859-1');

$p->setHandlers('Char' => $ch);
$p->parse($xml);

From perluniintro ...

...if your Perl script itself is encoded in UTF-8, you can use UTF-8 in your identifier names, and in string and regular expression literals, by saying use utf8. This is not the default because scripts with legacy 8-bit data in them would break.

regards.

[Anonymous Monk]

what does 'use utf8;' has to do with this problem? and sure, i tried this with no luck. try it yourself. the quoted text just means i can use utf-8 encoded variable names, like:

$Müller = 23;

[choroba]

Not really. Try:

perl -wE 'say length "Müller";
          use utf8;
          say length "Müller";'

[Anonymous Monk]

oh, i've overseen the

binmode STDOUT, ":encoding(UTF-8)";

line. well, this makes a difference. but i'm still puzzled, why i get utf-8 when i use ProtocolEncoding => 'ISO-8859-1' and which of those two ways does less math in encoding/decoding.

greetings

[remiah]

Maybe, you saved your script with utf-8 encoding. If you save the script as iso-8859-1, you will get iso-8859-1 result.

Below, 082.pl is utf-8 saved script and 082-1 is iso-8859-1 saved script."ü" is "c3 bc" in utf-8. "fc" in iso-8859-1.

>cat 082.pl |perl -ne 'print $1 if m!<word>(.*?)</word>!' | hd
00000000  4d c3 bc 6c 6c 65 72                              |M..ller|
00000007
>cat 082-1.pl |perl -ne 'print $1 if m!<word>(.*?)</word>!' | hd
00000000  4d fc 6c 6c 65 72                                 |M.ller|
00000006
>

[Anonymous Monk]

i benchmarked the binmode variant against the utf8 open variant down here. i made an xml file with 100 lines and 32000 ü's (utf8) in each line ((P)CDATA). the below script did it in 0.20 seconds while the 'use utf8; / binmode' method take about 17.5 seconds.

unfortunately perl crashes when i give a filehande to the parser while using the 'use open qw/:std :utf8/;' method when the file gets big. the 'use utf8; / binmode' method takes about 35 seconds when i pass the filehandle to the parser.

output got redirected to /dev/null

#!/usr/bin/perl

use XML::Parser;
#use utf8;
use open qw/:std :utf8/;

$ch = sub {
    my ($p, $w) = @_;
#    binmode STDOUT, ":encoding(UTF-8)";
    print "$w\n";
};

$p = XML::Parser->new(ProtocolEncoding => 'UTF-8');
$p->setHandlers('Char' => $ch);


my $xml = "";
open(F, '< x.xml');
while(<F>) { $xml .= $_; }

$p->parse($xml);
#$p->parse(*F);
close(F);