comment on

I don't see the problem here. You have \xC2\x96 (2 bytes) in your XML data, which is the correct UTF-8 encoding for U+0096, and indeed your code's output properly shows that:

0x73
0x75
0x72
0x65
0x20
0x96 <-- there it is
0x20
0x42
0x6C
0x61
0x63
0x6B
[download]

print "$str\n\n"; is a mistake, though. You shouldn't print unicode text data without specifying an output :encoding on the filehandle, or encode()ing it manually.

By the way, instead of the confusing, error-prone, and tedious process of figuring out the internal state of a variable using is_utf8 and a normal print, please use Devel::Peek instead. Its Dump function, called with your $str, would output:

SV = PV(0x8641d2c) at 0x82ea98c
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8)
  PV = 0x8631050 "sure \302\226 Black"\0 [UTF8 "sure \x{96} Black"]
  CUR = 13
  LEN = 16
[download]

As you can see, the "UTF8 flag" is on. The logical unicode string is within [UTF8 ... ], after the representation of the internal byte buffer.

Now, of course, the usefulness of the character U+0096, "START OF GUARDED AREA" is a rather different story. It probably is the result of mis-interpreting Windows-1251 data as ISO-8859-1 data. Windows-1251's 0x96 character is U+2013, "EN DASH", not U+0096.

In reply to Re: How to deal with malformed utf8 from XML parsing by Juerd
in thread How to deal with malformed utf8 from XML parsing by ribasushi

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Perl-Sensitive Sunglasses
	PerlMonks