Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

How do I determine encoding format of a file ?

by donno20 (Sexton)
on May 09, 2003 at 01:39 UTC ( #256728=categorized question: print w/replies, xml ) Need Help??
Contributed by donno20 on May 09, 2003 at 01:39 UTC
Q&A  > files

Answer: How do I determine encoding format of a file ?
contributed by graff

If you are pulling in HTML pages from the web, there will usually be a tag or attribute in the markup that specifies the intended character set. Grab some pages, inspect the HTML source, and see what those declarations look like.

If you are being handed "raw" (unmarked) text data, where there is absolutely no clue about what character encoding is being used, you need to have at least one of two things:

  • Correct knowledge about what language the text is written in.

    This will at least limit the number of possible encodings (e.g. Big5 verus GB2312 vs. GBK vs. unicode for Chinese), and if you become familiar with the different properties of established encodings for that language, working out logic to identify them is usually not a big problem.

  • Trained models of various encodings as applied to various languages.

    In particular, if you take samples of known data (where you're confident about the language and encoding), you can look at the frequency rankings or probabilities of byte n-grams -- usually bigrams will suffice, i.e. how many times each pairing of bytes occurs in a given set of data (the word "hello", in ASCII, has four bigrams: "he", "el", "ll", "lo").

If you don't know what language is being used in raw text, there may be statistics that would tell you whether it's likely to be using single-byte or multi-byte encoding, but from my perspective, this is a research question (I'm not that good at statistics).

Perl 5.8 has a module called "Encode::Guess", which might work well if you know the language involved and/or can provide some hints as to the likely candidates. (I haven't tried it yet, but it is admittedly limited and speculative at present.)

Answer: How do I determine encoding format of a file ?
contributed by idsfa

File::BOM provides get_encoding_from_filehandle and get_encoding from_stream to identify the encoding of Unicode files. Example:

use File::BOM qw( :all ); open $fh, '<', $filename; my ($encoding) = get_encoding_from_filehandle($fh);
Answer: How do I determine encoding format of a file ?
contributed by particle

have a look at File::MMagic, it guesses the filetype given the filename or a filehandle, and is quite configurable (you can add more file type descriptions based on regular expressions.) it's a handy little module.

Answer: How do I determine encoding format of a file ?
contributed by donno20

Read the first two bytes of the file. Corresponding encoding and hex codes are as follow:
unicode Little Endian = "\xFF\xFE"
unicode Big Endian = "\xFE\xFF"
utf8 = "\xEF\xBB"
ASCII = straight to content

Please (register and) log in if you wish to add an answer

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others drinking their drinks and smoking their pipes about the Monastery: (3)
    As of 2021-03-01 05:11 GMT
    Find Nodes?
      Voting Booth?

      No recent polls found