Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re^3: How to handle encodings?

by clinton (Priest)
on Mar 07, 2009 at 19:06 UTC ( #749078=note: print w/replies, xml ) Need Help??

in reply to Re^2: How to handle encodings?
in thread How to handle encodings?

A typical place where Encode::Guess falls down (through no fault of its own) is in differentiating one variant of iso-8859 from another.

Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

Without prior knowledge, you're up the creek without a paddle. So I agree wholeheartedly with Moritz's suggestion of converting everything to UTF8, while you still know what encoding it is in.

(graff - I know you're too wise a monk to have been suggesting otherwise, but I wanted to provide a simple example of just how limited Encode::Guess can be.)


Replies are listed 'Best First'.
Re^4: How to handle encodings?
by Narveson (Chaplain) on Mar 07, 2009 at 22:28 UTC
    Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

    Any moderately experienced human eye, given a paragraph of context. Have you ever played the game of pulling down the Character Encoding menu in your Web Browser and trying the same page in different encodings?

    This is not a dissent from moritz's and clinton's advice. By all means heed any available declarations.

    But in the absence of a declaration, the right encoding is humanly guessable, even when the alternatives are different members of the ISO-8859 series. And if it's humanly guessable, then a CPAN module should in principle be able to do it too.

    I don't know what algorithm Encode::Guess uses. My own algorithm (untested) would be

    • submit a phrase to Google as a search term;
    • accept Google's proposed spelling correction.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://749078]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2020-05-30 13:01 GMT
Find Nodes?
    Voting Booth?
    If programming languages were movie genres, Perl would be:

    Results (172 votes). Check out past polls.