|Problems? Is your data what you think it is?|
Matching/replacing a unicode character only works after decode()by FloydATC (Deacon)
|on Jul 25, 2014 at 10:10 UTC||Need Help??|
FloydATC has asked for the
wisdom of the Perl Monks concerning the following question:
After hours of struggling with a seemingly simple problem involving utf8 I finally made it work -- but I have no clue why. Allow me to explain.
I have a script which scrapes a particular web site for data about communication links. The data is utf-8 encoded and contains (among a great many other things) the unicode character 'GREEK SMALL LETTER MU' (U+03BC) which (after being scraped, put through several scripts, stored in MySQL, later extracted and presented on a web page) renders as "Î¼". My co-workers didn't really mind but after a couple of years it started to annoy me so much I reached the point where I just had to fix it. Today was that day.
I wanted to replace all occurences of this character with either unicode character 'MICRO SIGN' (U+00B5) which renders as expected, or (even better) simply with the HTML entity µ.
The method in question produces clickable links to present each commlink in many different contexts.
I knew the data stored in MySQL was utf8, the string was untoched and the web page charset was specified as utf8. If I tried to change it, the norwegian characters on the same page would become garbled so I knew the encoding setting was sent and detected properly.
So... utf8 in, no encoding/decoding or string mangling prior to the regex... and still the regex didn't match.
My question is... Why?! Before decoding the utf8 string, how could the string go from input to output unchanged but fail to match the regex? Why do I need to decode the utf8 string to match an utf8 character when the string already prints as an utf8 character? This is so confusing...UPDATE:
This is exactly why I have always hated Unicode. Why, oh why could I not have left this stupid bug alone.
Time flies when you don't know what you're doing