Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re^3: How to deal with malformed utf8 from XML parsing

by Juerd (Abbot)
on Jan 09, 2008 at 22:11 UTC ( #661521=note: print w/replies, xml ) Need Help??

in reply to Re^2: How to deal with malformed utf8 from XML parsing
in thread How to deal with malformed utf8 from XML parsing

I thought I can print unicode to STDOUT. I will read more on that.

See perlunitut. Filehandles work with bytes, not characters.

If you pay closer attention you will see that I am using the validating capability of is_utf8 ($string, 'true_value').

It checks the INTERNAL BYTE BUFFER of the unicode string. It is an internal consistency check, and should only be used to verify Perl's internal functioning, not your own strings. Apparently the [INTERNAL] in the documentation is not clear enough, given the huge number of people who don't realise that it is an internal function. I'll see if I can get that changed.

Juerd # { site => '', do_not_use => 'spamtrap', perl6_server => 'feather' }

Replies are listed 'Best First'.
Re^4: How to deal with malformed utf8 from XML parsing
by ribasushi (Monk) on Jan 09, 2008 at 23:31 UTC
    Well today was definitely a fruitful day. I learned about C0/C1 control codes which were the reason for google to complain. I also realized where this stuff actually comes from (someone pasting a mis-encoded chunk of text into a browser window). Finally I know not to use is_utf8 anymore :)
    Thank you for your comments.

    P.S.How I ended up fixing this:
    $_ =~ s/[\x{80}-\x{9F}]/\x{FFFD}/g;

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://661521]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2018-01-19 21:47 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (223 votes). Check out past polls.