Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

XML::XPath and character encoding

by mboudreau (Acolyte)
on Dec 29, 2015 at 16:10 UTC ( [id://1151377]=perlquestion: print w/replies, xml ) Need Help??

mboudreau has asked for the wisdom of the Perl Monks concerning the following question:

I've got a fairly old script that uses XML::XPath to parse an XML file that includes personal names with accented letters, e.g.,

<?xml version="1.0" encoding="UTF-8"?> <contrib contrib-type="author"> <name> <surname>San José Estépar</surname> <given-names>Raul</given-names> </name> </contrib>

The names are assigned variables and then inserted into a MySQL database. Later, a different script collects the names from the database and writes them to a new XML file.

The problem (which may have been going on for a while and has only just been reported) is that the accented letters aren't surviving the journey. When I view the names in the database (using Sequel Pro 1.1 for OS X), I see the correct characters. But when the names are written to a new XML file, the characters appear as question marks (viewing the XML file in Oxygen XML Editor 17.0 for OS X or just via the Unix 'more' command from the OS X Terminal app).

I'm pretty confident that the source XML file is OK. It has the explicit XML declaration (as in the sample above) including 'encoding="UTF-8"', and it displays correctly any way I view it (from the Unix command-line, from Oxygen, etc.).

I have also verified that reading the XML file and writing it out again like so:

my $xml; open (SOURCE, $source_file) or die "ERROR: Cannot open filehandle to read $source_file: $!\n"; { local $/; $xml = <SOURCE>; } close(SOURCE) or die "ERROR: Cannot close filehandle on $source_file: $!\n"; print $xml;

preserves the characters. The problem occurs when I parse the file content like so:

my $xp = XML::XPath->new(xml => $xml) or die "ERROR: XML::XPath cannot parse target file: $!\n"; my $contrib_nodeset = $xp->find('/contrib[@contrib-type="author"]'); foreach my $contrib_node ($contrib_nodeset->get_nodelist) { my $given_names = $xp->find('./name/given-names', $contrib_node); my $surname = $xp->find('./name/surname', $contrib_node); print "CONTRIB: $given_names $surname\n"; }

I confess I haven't had to deal much with character encoding before, so I'd be grateful for any advice on how to troubleshoot this problem.

Replies are listed 'Best First'.
Re: XML::XPath and character encoding
by Anonymous Monk on Dec 29, 2015 at 16:31 UTC
    I have also verified that reading the XML file and writing it out again like so:
    Probably string downgrading issue again. Try to
    binmode STDOUT, ':encoding(utf-8)';
    put that on top of your program.

    If that doesn't work, post the output of

    my $xp = XML::XPath->new(xml => $xml) or die "ERROR: XML::XPath cannot parse target file: $!\n"; my $contrib_nodeset = $xp->find('/contrib[@contrib-type="author"]'); foreach my $contrib_node ($contrib_nodeset->get_nodelist) { my $given_names = $xp->find('./name/given-names', $contrib_node); my $surname = $xp->find('./name/surname', $contrib_node); printf "given_names => (%d) %vx\nsurname => (%d) %vx\n", utf8::is_utf8($given_names), $given_names, utf8::is_utf8($surname), $surname; }
    (with problematic strings)
      Thank you! That worked perfectly.
Re: XML::XPath and character encoding
by Anonymous Monk on Dec 29, 2015 at 18:01 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1151377]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others studying the Monastery: (4)
As of 2024-03-29 12:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found