http://www.perlmonks.org?node_id=921975


in reply to Can't tell if UTF-8... or just binary...

This makes no sense:
use utf8;
utf8_decode("My favorite pokemon is ブラッキー")

«use utf8;» will decode, and then you try to decode again?

Contrary to what you say, @valid does not contain valid UTF-8 strings, since you remove the UTF-8 encoding.


I'm trying to find a way to determine whether some arbitrary blob of data is text or just binary.

If there are any characters above 255, then it's surely text. Beyond that, it's impossible to determine.

$ perl -MDevel::Peek -e'$_="\x{00C9}ric"; Dump($_);' FLAGS = (POK,pPOK) PV = 0x826d060 "\311ric"\0 # Text (My name) $ perl -MDevel::Peek -e'$_=pack("N", 0xC9726963); Dump($_);' FLAGS = (POK,pPOK) PV = 0xa16b2d8 "\311ric"\0 # Binary

(Irrelevant output removed for brevity and clarity.)

figure out a way to detect whether data is UTF-8 or just random binary:

If there are any characters above 127, just try to decode it. If you're successful, it's surely UTF-8.

my $is_utf8 = eval { decode('UTF-8', $bytes, Encode::FB_CROAK|Encode::LEAVE_SRC); 1 };

The chances of getting a false positive are extremely slim. However, if all the characters are below 128, it's impossible to determine.

$ perl -MDevel::Peek -e'$_="ABCD"; Dump($_);' FLAGS = (POK,pPOK) PV = 0x81bb060 "ABCD"\0 # Text $ perl -MDevel::Peek -e'$_=pack("N", 0x41424344); Dump($_);' FLAGS = (POK,pPOK) PV = 0x9e0a2c8 "ABCD"\0 # Binary