After hours of struggling with a seemingly simple problem involving utf8 I finally made it work -- but I have no clue why. Allow me to explain.

I have a script which scrapes a particular web site for data about communication links. The data is utf-8 encoded and contains (among a great many other things) the unicode character 'GREEK SMALL LETTER MU' (U+03BC) which (after being scraped, put through several scripts, stored in MySQL, later extracted and presented on a web page) renders as "μ". My co-workers didn't really mind but after a couple of years it started to annoy me so much I reached the point where I just had to fix it. Today was that day.

I wanted to replace all occurences of this character with either unicode character 'MICRO SIGN' (U+00B5) which renders as expected, or (even better) simply with the HTML entity µ.

The method in question produces clickable links to present each commlink in many different contexts.

sub commlink {
  my $self = shift;
  return "" unless $self->{'id'};
  my $label = $self->{'label'};
  $label =~ s/\x{00b5}/\µ/g;
  $label =~ s/\x{03bc}/\μ/g; # Looks almost exactly the same as &mi
+cro;
  return "<A href=\"commlink.html?id=".$self->{'id'}."\" class=\"".$se
+lf->{'state'}."\">".$label."</A>";
}
[download]

I knew the data stored in MySQL was utf8, the string was untoched and the web page charset was specified as utf8. If I tried to change it, the norwegian characters on the same page would become garbled so I knew the encoding setting was sent and detected properly.

So... utf8 in, no encoding/decoding or string mangling prior to the regex... and still the regex didn't match.

The solution?

sub commlink {
  my $self = shift;
  return "" unless $self->{'id'};
  my $label = $self->{'label'};
  $label = decode('utf8', $label); # Why? It's already utf8 and I need
+ it to stay utf8
  $label =~ s/\x{00b5}/\µ/g;
  $label =~ s/\x{03bc}/\μ/g; # Looks almost exactly the same as &mi
+cro;
  return "<A href=\"commlink.html?id=".$self->{'id'}."\" class=\"".$se
+lf->{'state'}."\">".$label."</A>";
}
[download]

My question is... Why?! Before decoding the utf8 string, how could the string go from input to output unchanged but fail to match the regex? Why do I need to decode the utf8 string to match an utf8 character when the string already prints as an utf8 character? This is so confusing...

OK, thanks for the pointers. It sounds so very very simple in theory, but in practice... This system is made up from more than 50 different scripts and modules that shuffle data back and forth and present it via HTML, SVG, generates javascript, text messages, emails and what have you. After I started trying to fix things to "do it right" then absolutely everything broke. I'm going to need weeks to get on top of this.

This is exactly why I have always hated Unicode. Why, oh why could I not have left this stupid bug alone.