Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Can't tell if UTF-8... or just binary...

by ikegami (Pope)
on Aug 23, 2011 at 19:30 UTC ( #921975=note: print w/replies, xml ) Need Help??


in reply to Can't tell if UTF-8... or just binary...

This makes no sense:
use utf8;
utf8_decode("My favorite pokemon is ブラッキー")

«use utf8;» will decode, and then you try to decode again?

Contrary to what you say, @valid does not contain valid UTF-8 strings, since you remove the UTF-8 encoding.


I'm trying to find a way to determine whether some arbitrary blob of data is text or just binary.

If there are any characters above 255, then it's surely text. Beyond that, it's impossible to determine.

$ perl -MDevel::Peek -e'$_="\x{00C9}ric"; Dump($_);' FLAGS = (POK,pPOK) PV = 0x826d060 "\311ric"\0 # Text (My name) $ perl -MDevel::Peek -e'$_=pack("N", 0xC9726963); Dump($_);' FLAGS = (POK,pPOK) PV = 0xa16b2d8 "\311ric"\0 # Binary

(Irrelevant output removed for brevity and clarity.)

figure out a way to detect whether data is UTF-8 or just random binary:

If there are any characters above 127, just try to decode it. If you're successful, it's surely UTF-8.

my $is_utf8 = eval { decode('UTF-8', $bytes, Encode::FB_CROAK|Encode::LEAVE_SRC); 1 };

The chances of getting a false positive are extremely slim. However, if all the characters are below 128, it's impossible to determine.

$ perl -MDevel::Peek -e'$_="ABCD"; Dump($_);' FLAGS = (POK,pPOK) PV = 0x81bb060 "ABCD"\0 # Text $ perl -MDevel::Peek -e'$_=pack("N", 0x41424344); Dump($_);' FLAGS = (POK,pPOK) PV = 0x9e0a2c8 "ABCD"\0 # Binary

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://921975]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (3)
As of 2021-10-19 11:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My first memorable Perl project was:







    Results (76 votes). Check out past polls.

    Notices?