|Problems? Is your data what you think it is?|
Does the script use Encode::Guess?
Good point. I had forgotten about that module. Will look into that.
What does it do exactly?
With Encode: $string = decode("utf8", $string,FB_DEFAULT )
... cleaning up encodings is actually a task that is very difficult to automate in general.
I agree. But since XML is very picky about encoding errors, and the XML feed must continue in some way, this is (for now) the solution.
It's a problem that many people (will be|are) facing when migrating legacy systems to Unicode/XML aware systems, which is why I think it warrants someting on CPAN.