http://www.perlmonks.org?node_id=197126


in reply to XML::Parser Encoding (UTF-8 -> ISO-8859-1)

I'm not an expert on Text::Iconv (I prefer Unicode::String, it seems to be more portable) but I know a little about Unicode encodings.

A couple of notes, some of which may be relevant. Forgive me if you know all this -- I figured it might be useful to somebody who's looking for this kind of information, even if it doesn't necessarily help Emanuel:

  1. ISO-8859-1 doesn't have the richness to encode every possible character from UTF-8. Many Eastern European characters (not to mention South and East Asian characters) cannot be encoded in ISO-8859-1. There just aren't enough bits. Are there characters in your data outside the range (U+0000 .. U+00FF) ?
  2. Perl's internals are in UTF-8, so the fact that print outputs correct-looking data may be because Text::Iconv is not doing it's job correctly (or you're not using the feature the way it's intended). In other words, if your data is *still* in UTF-8, then it will probably print correctly.
  3. If you're using .nix or cygwin, then you probably have the od (octal dump) tool available, which I find indispensable for determining codeset issues (editors like vi and emacs tend to operate at too high a level, because they try to interpret the encoding for you and things "look fine" even when they're in the wrong encoding). I use:od -a all the time to figure out whether I've used encoding tools correctly.

HTH, jkahn