http://www.perlmonks.org?node_id=998901

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I'm just a bit confused by the behaving of the XML::Parser. I have an UTF-8 encoded string in $xml (Müller in the (P)CDATA section) and I like to get it in UTF-8 on the output. 

#!/usr/bin/perl

use XML::Parser;
$xml = "<word>Müller</word>";

$ch = sub {
    my ($p, $w) = @_;
    print "$w\n";
};

# the next commented 2 lines mean the same
# and translate the output to iso-8859-1
# $p = XML::Parser->new();
# $p = XML::Parser->new(ProtocolEncoding => 'UTF-8');

# this line do the right job, but why??
$p = XML::Parser->new(ProtocolEncoding => 'ISO-8859-1');

$p->setHandlers('Char' => $ch);
$p->parse($xml);

[download]
Please let me understand why I need to set ProtocolEncoding to 'ISO-8859-1' to get the UTF-8 encoded data as UTF-8 and not as iso-8859-1. Also let me know if this will first translate the data to iso-8859-1 and then back to UTF-8.

Disclaimer: It's not my decision to use XML::Parser, even if i like/dislike it. It's used by an module and it's out of discussion if this or that xml parser is better or not.

thank you so much & hear my prayer

Replies are listed 'Best First'.
Re: UTF-8 and XML::Parser
by remiah (Hermit) on Oct 14, 2012 at 02:35 UTC

    Hello.

    You need "use utf8;" for literal strings to be treated as utf8 "character". 

    #!/usr/bin/perl

use XML::Parser;
use utf8;           ###added
$xml = "<word>Müller</word>";

$ch = sub {
my ($p, $w) = @_;
    binmode STDOUT, ":encoding(UTF-8)"; ###added
    print "#$w#\n";
};

# the next commented 2 lines mean the same
# and translate the output to iso-8859-1
# $p = XML::Parser->new();
$p = XML::Parser->new(ProtocolEncoding => 'UTF-8');

# this line do the right job, but why??
#$p = XML::Parser->new(ProtocolEncoding => 'ISO-8859-1');

$p->setHandlers('Char' => $ch);
$p->parse($xml);

    [download]
    From perluniintro ...

    ...if your Perl script itself is encoded in UTF-8, you can use UTF-8 in your identifier names, and in string and regular expression literals, by saying use utf8. This is not the default because scripts with legacy 8-bit data in them would break.

    regards.

[reply]
[d/l]
      oh, i've overseen the 
      binmode STDOUT, ":encoding(UTF-8)";

      [download]
      line. well, this makes a difference. but i'm still puzzled, why i get utf-8 when i use ProtocolEncoding => 'ISO-8859-1' and which of those two ways does less math in encoding/decoding.

      greetings

[reply]
[d/l]

        Maybe, you saved your script with utf-8 encoding. If you save the script as iso-8859-1, you will get iso-8859-1 result.

        Below, 082.pl is utf-8 saved script and 082-1 is iso-8859-1 saved script."ü" is "c3 bc" in utf-8. "fc" in iso-8859-1. 

        >cat 082.pl |perl -ne 'print $1 if m!<word>(.*?)</word>!' | hd
00000000  4d c3 bc 6c 6c 65 72                              |M..ller|
00000007
>cat 082-1.pl |perl -ne 'print $1 if m!<word>(.*?)</word>!' | hd
00000000  4d fc 6c 6c 65 72                                 |M.ller|
00000006
>

        [download]

[reply]
[d/l]
      what does 'use utf8;' has to do with this problem? and sure, i tried this with no luck. try it yourself. the quoted text just means i can use utf-8 encoded variable names, like: 
      $Müller = 23;

      [download]
[reply]
[d/l]
        Not really. Try: 
        perl -wE 'say length "Müller";
          use utf8;
          say length "Müller";'

        [download]
        لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
[reply]
[d/l]
Re: UTF-8 and XML::Parser
by grantm (Parson) on Oct 15, 2012 at 20:48 UTC

    All of the XML parser modules expect raw bytes of XML as input. Therefore your results may differ if you parse from a file or open filehandle rather than from a string - it all depends on how the data got into the string.

    If you pass a string of XML to XML::Parser you need to be sure that it is a byte string and not a character string. So the anonymous monk's suggestion to 'use utf8;' is exactly the wrong thing in this case - it would convert all non-ASCII literal strings in your script to Perl's internal character string representation. To convert from that to a form that an XML parser can read you'd need to use something like:

      my $bytes = Encode::encode_utf8($string);

    [download]

    Perl's internal character string representation is similar to but not exactly the same as UTF8. In particular, some characters in the range U+0080 to U+00FF are represented as a single byte (the ISO8859-1 form) instead of the 2 bytes you'd expect from UTF8.

[reply]
[d/l]

Back to Seekers of Perl Wisdom