Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^3: length() miscounting UTF8 characters?

by farang (Hermit)
on Apr 28, 2014 at 14:28 UTC ( #1084130=note: print w/ replies, xml ) Need Help??


in reply to Re^2: length() miscounting UTF8 characters?
in thread length() miscounting UTF8 characters?

the perldoc entry for length (which I checked beforehand to make sure it wouldn't count bytes -- hence my confusion)
It "normally deals in logical characters", but its logic doesn't cover all the intricacies of unicode.

Do you have any specific languages or complex data in mind with which it might fail?
Yes, Thai language is the main one I'm involved with. The modified script below shows that length counts diacriticals in Thai, which may or may not be what is wanted, and is inconsistent with the results for Latin diacriticals in your dataset, which length isn't counting separately. I'm using pre tags so that the Thai will display correctly and shortened lines to facilitate copy/paste.
#!/usr/bin/env perl
use warnings;
use v5.14;
use Unicode::Normalize qw/NFD/;
binmode STDOUT, 'utf8';
binmode DATA, 'encoding(utf-8)';
 
while (<DATA>) {
    chomp;
    print $_, ': ';
    s/[A-Za-z]//g;
    my $alphacount = () = /\p{Alpha}/g;
    say "non-(A-Za-z) symbols <$_>", 
        " contain $alphacount", 
        " alphabetic characters and ",
        getdia($_), " diacritical chars.";
    say "length() thinks there are ", 
        length, " characters\n";
}

sub getdia {
    my $normalized = NFD($_[0]);
    my $diacount = () = 
        $normalized =~ /\p{Dia}/g;
    return $diacount;
}

__DATA__
เป็น
ผู้หญิง
เมื่อวันก่อน
Š­aber
Š­ahn˙tur
Š­ak÷lkun


Comment on Re^3: length() miscounting UTF8 characters?
Re^4: length() miscounting UTF8 characters?
by AppleFritter (Curate) on Apr 28, 2014 at 19:37 UTC

    Intricacies - that's putting it mildly! Right now it feels like the more I learn about Unicode, the less I know. ;)

    I'll make a mental note to avoid length when dealing with Unicode data, and to normalize strings before working with them.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1084130]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (8)
As of 2014-12-19 06:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (72 votes), past polls