help needed in file encoding

[uva]

dear monks,
i can find whether the particular string is utf8 or not by means of utf8::is_utf8();
but how to detect which encoding is used for the particular file. in order to check if the file is encoded in utf8 or utf16 or someother format.

Edited by planetscape - fixed br tags

[zentara]

Maybe Encoding Detection will help you?

[ikegami]

You could check if the first few bytes contains a BOM.

if    (substr($text, 0, 4) eq "\x00\x00\xFE\xFF") { utf32be }
elsif (substr($text, 0, 4) eq "\xFF\xFE\x00\x00") { utf32le }
elsif (substr($text, 0, 2) eq "\xFE\xFF"        ) { utf16be }
elsif (substr($text, 0, 2) eq "\xFF\xFE"        ) { utf16le }
elsif (substr($text, 0, 3) eq "\xEF\xBB\xBF"    ) { utf8    }
else {
   No BOM found. Might not be UTF.
   Use another method to guess the
   encoding.
}

It's not very reliable. The protocol which encases your stream/data must really specify the encoding for things to work smoothly.

[duckyd]

Consider File::BOM if you need to do something like this.

[uva]

i tried with utf8::is_utf8($string) ,the $string is got from the big5 encoded file,i got the output as 1(true). how does it happen . Is all internal representation is utf8 in windows?.

[timos]

In Linux: wc -m tells you the number of characters, wc -c the numer of bytes. If the last one is greater then the first one it is probably UTF-8/16 encoded.

[uva]

hai Zentara,
http://www.cis.upenn.edu/~wjposer/.downloads/unidesc.tgz
this link is not going , giving me as "not found" error.

hai timos
i am using windows 2000, is there any possibility in fining in windows.

[Anonymous Monk]

See Encode::Guess, Lingua::DetectCharset, Lingua::ZH::HanDetect or CharsetToolkit (Java), Mozilla Charset Detectors.

Tools: unidesc, mguesser, enca. You might be able to compile them for Windows with MinGW or Cygwin. Hey, CamelPack comes with MinGW and a nice frontend!