Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

UTF-8 and XML::Parser

by Anonymous Monk
on Oct 14, 2012 at 02:06 UTC ( #998901=perlquestion: print w/ replies, xml ) Need Help??
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I'm just a bit confused by the behaving of the XML::Parser. I have an UTF-8 encoded string in $xml (Müller in the (P)CDATA section) and I like to get it in UTF-8 on the output.

#!/usr/bin/perl use XML::Parser; $xml = "<word>Müller</word>"; $ch = sub { my ($p, $w) = @_; print "$w\n"; }; # the next commented 2 lines mean the same # and translate the output to iso-8859-1 # $p = XML::Parser->new(); # $p = XML::Parser->new(ProtocolEncoding => 'UTF-8'); # this line do the right job, but why?? $p = XML::Parser->new(ProtocolEncoding => 'ISO-8859-1'); $p->setHandlers('Char' => $ch); $p->parse($xml);
Please let me understand why I need to set ProtocolEncoding to 'ISO-8859-1' to get the UTF-8 encoded data as UTF-8 and not as iso-8859-1. Also let me know if this will first translate the data to iso-8859-1 and then back to UTF-8.

Disclaimer: It's not my decission to use XML::Parser, even if i like/dislike it. It's used by an module and it's out of discussion if this or that xml parser is better or not.

thank you so much & hear my prayer

Comment on UTF-8 and XML::Parser
Download Code
Re: UTF-8 and XML::Parser
by remiah (Hermit) on Oct 14, 2012 at 02:35 UTC

    Hello.

    You need "use utf8;" for literal strings to be treated as utf8 "character".

    #!/usr/bin/perl use XML::Parser; use utf8; ###added $xml = "<word>Müller</word>"; $ch = sub { my ($p, $w) = @_; binmode STDOUT, ":encoding(UTF-8)"; ###added print "#$w#\n"; }; # the next commented 2 lines mean the same # and translate the output to iso-8859-1 # $p = XML::Parser->new(); $p = XML::Parser->new(ProtocolEncoding => 'UTF-8'); # this line do the right job, but why?? #$p = XML::Parser->new(ProtocolEncoding => 'ISO-8859-1'); $p->setHandlers('Char' => $ch); $p->parse($xml);
    From perluniintro ...

    ...if your Perl script itself is encoded in UTF-8, you can use UTF-8 in your identifier names, and in string and regular expression literals, by saying use utf8. This is not the default because scripts with legacy 8-bit data in them would break.

    regards.

      what does 'use utf8;' has to do with this problem? and sure, i tried this with no luck. try it yourself. the quoted text just means i can use utf-8 encoded variable names, like:
      $Müller = 23;
        Not really. Try:
        perl -wE 'say length "Müller"; use utf8; say length "Müller";'
        لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
      oh, i've overseen the
      binmode STDOUT, ":encoding(UTF-8)";
      line. well, this makes a difference. but i'm still puzzled, why i get utf-8 when i use ProtocolEncoding => 'ISO-8859-1' and which of those two ways does less math in encoding/decoding.

      greetings

        Maybe, you saved your script with utf-8 encoding. If you save the script as iso-8859-1, you will get iso-8859-1 result.

        Below, 082.pl is utf-8 saved script and 082-1 is iso-8859-1 saved script."ü" is "c3 bc" in utf-8. "fc" in iso-8859-1.

        >cat 082.pl |perl -ne 'print $1 if m!<word>(.*?)</word>!' | hd 00000000 4d c3 bc 6c 6c 65 72 |M..ller| 00000007 >cat 082-1.pl |perl -ne 'print $1 if m!<word>(.*?)</word>!' | hd 00000000 4d fc 6c 6c 65 72 |M.ller| 00000006 >

Re: UTF-8 and XML::Parser
by grantm (Parson) on Oct 15, 2012 at 20:48 UTC

    All of the XML parser modules expect raw bytes of XML as input. Therefore your results may differ if you parse from a file or open filehandle rather than from a string - it all depends on how the data got into the string.

    If you pass a string of XML to XML::Parser you need to be sure that it is a byte string and not a character string. So the anonymous monk's suggestion to 'use utf8;' is exactly the wrong thing in this case - it would convert all non-ASCII literal strings in your script to Perl's internal character string representation. To convert from that to a form that an XML parser can read you'd need to use something like:

    my $bytes = Encode::encode_utf8($string);

    Perl's internal character string representation is similar to but not exactly the same as UTF8. In particular, some characters in the range U+0080 to U+00FF are represented as a single byte (the ISO8859-1 form) instead of the 2 bytes you'd expect from UTF8.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://998901]
Approved by Athanasius
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2014-10-01 01:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (386 votes), past polls