Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re: Unicode nightmare: is there a cheat sheet or solutions diagram?

by mirod (Canon)
on Jun 10, 2010 at 13:44 UTC ( [id://844006]=note: print w/replies, xml ) Need Help??


in reply to Unicode nightmare: is there a cheat sheet or solutions diagram?

If that's any comfort, you are probably not the only one to have been tortured by Unicode for that long. I have a last-resort shell trick that goes  iconv -f utf8 -t utf8 <dodgy_file> || iconv -f iso-8859-1 -t utf8 <dodgy_file> > <utf8_file> for those cases where I am not 100% sure of the encoding I get, and I don't have the time to debug it properly.

In your case I wonder if the problem is not that you have some non Latin1 characters in your input (that got there through entities in the HTML maybe, something like &mdash; for example.

My machine has got UTF-8 locale, so I can't reproduce exactly your problem, but the 2 one liners below give me outputs in different encodings:

perl -MHTML::TreeBuilder::XPath -e'$t= HTML::TreeBuilder::XPath->new; $t->parse( "<html><body><p>para &copy;</p></body></html>"); $p=$t->findvalue( "//p"); print $p, "\n";'

That first one gives me a result in latin1.

perl -MHTML::TreeBuilder::XPath -e'$t= HTML::TreeBuilder::XPath->new; $t->parse( "<html><body><p>para &copy; &mdash;</p></body></html>"); @p=$t->findnodes( "//p"); print $p[0]->as_text(), "\n";'

I just added an &mdash;, an entity that cannot be printed in Latin1, to the input et voilą! Now the result is in UTF-8.

In that case using HTML::Entities should help:

perl -MHTML::TreeBuilder::XPath -MHTML::Entities -e' $t= HTML::TreeBuilder::XPath->new; $t->parse( "<html><body><p>para &copy; &mdash;</p></body></html>"); @p=$t->findnodes( "//p"); $out= $p[0]->as_text; encode_entities( $out); print $out, "\n";'

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://844006]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2024-03-19 02:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found