How best to avoid mojibake, when attempting to automatically convert documents to utf-8?by taint (Chaplain)
|on Dec 20, 2013 at 23:22 UTC||Need Help??|
taint has asked for the
wisdom of the Perl Monks concerning the following question:
This is a first step in the creation of an All-To-UTF-8 type Perl module. I've spent some time researching the possibility of using something already available to accomplish this task -- plow through a directory of text documents of UNknown encoding/charset recursively, and accurately detect their encoding/charset, and reliably convert them to utf-8.
Yes. But what if their not correctly defined, in the first place?
While I recognize that Perl is said to be (very?) good at handling this sort of thing. My personal experience, and judging by other nodes here at PM, indicates there is room for improvement. It is not my intention to "fix Perl" in any way. But rather, create a module that will "ease the pain" often encountered doing such things.
This is a formal solicitation for comments, examples, and recommendations
for accomplishing this task. Thus far, I have carefully looked at
Encode::Unicode -- OH, and
Bush hid the facts
Well. In summary. At this point, I'm attempting to find the best solution to detecting, in the best possible manner, the current/actual encoding/charset of a document. Methods for the actual conversion will come in a different node/thread. At a later time. :)
Thank you for all your consideration.
--ChrisUPDATE: Just a thanks to Your Mother, and moritz. For some hints
Yes. What say about me, is true.