Re: How to deal with malformed utf8 from XML parsing

[Juerd]

I don't see the problem here. You have \xC2\x96 (2 bytes) in your XML data, which is the correct UTF-8 encoding for U+0096, and indeed your code's output properly shows that:

0x73
0x75
0x72
0x65
0x20
0x96 <-- there it is
0x20
0x42
0x6C
0x61
0x63
0x6B

print "$str\n\n"; is a mistake, though. You shouldn't print unicode text data without specifying an output :encoding on the filehandle, or encode()ing it manually.

By the way, instead of the confusing, error-prone, and tedious process of figuring out the internal state of a variable using is_utf8 and a normal print, please use Devel::Peek instead. Its Dump function, called with your $str, would output:

SV = PV(0x8641d2c) at 0x82ea98c
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8)
  PV = 0x8631050 "sure \302\226 Black"\0 [UTF8 "sure \x{96} Black"]
  CUR = 13
  LEN = 16

As you can see, the "UTF8 flag" is on. The logical unicode string is within [UTF8 ... ], after the representation of the internal byte buffer.

Now, of course, the usefulness of the character U+0096, "START OF GUARDED AREA" is a rather different story. It probably is the result of mis-interpreting Windows-1251 data as ISO-8859-1 data. Windows-1251's 0x96 character is U+2013, "EN DASH", not U+0096.

[ribasushi]

print "$str\n\n"; is a mistake, though. You shouldn't print unicode text data without specifying an output :encoding on the filehandle, or encode()ing it manually.

I thought I can print unicode to STDOUT. I will read more on that.


By the way, instead of the confusing, error-prone, and tedious process of figuring out the internal state of a variable using is_utf8 and a normal print, please use Devel::Peek instead

If you pay closer attention you will see that I am using the validating capability of is_utf8 ($string, 'true_value').


I verified your claim, indeed it seems that this is not malformed utf. The reason I started digging is google complaining this is not a valid character. I will troubleshoot more.

Thank you!

[Juerd]

I thought I can print unicode to STDOUT. I will read more on that.

See perlunitut. Filehandles work with bytes, not characters.

If you pay closer attention you will see that I am using the validating capability of is_utf8 ($string, 'true_value').

It checks the INTERNAL BYTE BUFFER of the unicode string. It is an internal consistency check, and should only be used to verify Perl's internal functioning, not your own strings. Apparently the [INTERNAL] in the documentation is not clear enough, given the huge number of people who don't realise that it is an internal function. I'll see if I can get that changed.

[ribasushi]

Well today was definitely a fruitful day. I learned about C0/C1 control codes which were the reason for google to complain. I also realized where this stuff actually comes from (someone pasting a mis-encoded chunk of text into a browser window). Finally I know not to use is_utf8 anymore :)
Thank you for your comments.

P.S.How I ended up fixing this:

$_ =~ s/[\x{80}-\x{9F}]/\x{FFFD}/g;