Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^3: How to handle encodings?

by clinton (Priest)
on Mar 07, 2009 at 19:06 UTC ( #749078=note: print w/ replies, xml ) Need Help??


in reply to Re^2: How to handle encodings?
in thread How to handle encodings?

A typical place where Encode::Guess falls down (through no fault of its own) is in differentiating one variant of iso-8859 from another.

Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

Without prior knowledge, you're up the creek without a paddle. So I agree wholeheartedly with Moritz's suggestion of converting everything to UTF8, while you still know what encoding it is in.

(graff - I know you're too wise a monk to have been suggesting otherwise, but I wanted to provide a simple example of just how limited Encode::Guess can be.)

Clint


Comment on Re^3: How to handle encodings?
Download Code
Re^4: How to handle encodings?
by Narveson (Chaplain) on Mar 07, 2009 at 22:28 UTC
    Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

    Any moderately experienced human eye, given a paragraph of context. Have you ever played the game of pulling down the Character Encoding menu in your Web Browser and trying the same page in different encodings?

    This is not a dissent from moritz's and clinton's advice. By all means heed any available declarations.

    But in the absence of a declaration, the right encoding is humanly guessable, even when the alternatives are different members of the ISO-8859 series. And if it's humanly guessable, then a CPAN module should in principle be able to do it too.

    I don't know what algorithm Encode::Guess uses. My own algorithm (untested) would be

    • submit a phrase to Google as a search term;
    • accept Google's proposed spelling correction.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://749078]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (18)
As of 2014-07-10 19:03 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    When choosing user names for websites, I prefer to use:








    Results (215 votes), past polls