Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^2: length() miscounting UTF8 characters?

by moritz (Cardinal)
on Apr 28, 2014 at 19:27 UTC ( #1084181=note: print w/ replies, xml ) Need Help??


in reply to Re: length() miscounting UTF8 characters?
in thread length() miscounting UTF8 characters?

Using the length function to count unicode characters is a bug waiting to happen.

Well, all perl builtins work at the codepoint level, including length. Depending on your definition of "character", that might or might not be what the OP wants.

I've attempted to implement "extended grapheme cluster" (that is, any base char + modifiers is considered a "character") logic in Perl6::Str. Feedback very welcome :-).


Comment on Re^2: length() miscounting UTF8 characters?
Re^3: length() miscounting UTF8 characters?
by AppleFritter (Chaplain) on Apr 28, 2014 at 19:47 UTC

    Yes, "extended grapheme clusters" are what I'm apparently interested and what I'd ordinarily call "characters", rather than codepoints.

    I've not looked at Perl 6 yet, but being able to work with Unicode data from a high-level perspective, without caring too much about implementation details such as the various representation layers (the encoding layer that take bytes to codepoints, and then the next one that takes codepoints to "extended grapheme clusters") would be a huge boon for many, including me.

      Representing the written languages of the world on computers is complex. The Unicode Standard is complex. Programming Unicode is complex. There's a limit to how much of this complexity can be hidden from computer programmers.

      If you want to understand Unicode better, and how to think correctly about programming Unicode, read Tom Christiansen's excellent Stack Overflow post here. If, after reading his well-known post, you find you need more of Tom's Perl Unicode wisdom, then come back to PerlMonks and read what tchrist has written about the topic here.

        OK, thanks again. Whoa, that's a lot of information to digest. I'll have to meditate on this, as it were.
      You seem to be thinking that Moritz is suggesting you use or look in to Perl 6. I'm pretty sure he is not.

      Yes, Perl 6 does aim to enable working from the high-level perspective you describe.

      No, it's not worth you looking in to Perl 6 if you just want to get stuff done and don't care to have fun figuring out how it works and contributing to the Perl 6 effort by fixing and working around bugs and speed problems etc.

      Moritz mentioned Perl6::Str. This is a perl 5 module he wrote that must be used by perl 5 scripts. It is one of several perl (5) modules written by perl 5 programmers who chose to use the Perl6 namespace in CPAN to reflect the module being somehow related to Perl 6.

      To quote the Perl6::Str module description:

      Perl 5 offers string manipulation at the byte level (for non-upgraded strings) and at the codepoint level (for decoded strings). However it fails to provide string manipulation at the grapheme level, that is it has no easy way of treating a sequence of codepoints, in which all but the first are combining characters (like accents, for example) as one character.

      Perl6::Str tries to solve this problem by introducing a string object with an API similar to that of Perl 6 (as far as possible), and emulating common operations such as substr, chomp and chop at the grapheme level. It also introduces builtin string methods found in Perl 6 such as samecase.

      In summary, aiui: for production grade grapheme level handling, it's best to rely on perl 5, accepting all the intricacies and complications that inevitably arise; for some Perl 6 like features in perl 5 that attempt to hide some of that complexity (in trade for some loss of some magic and speed) you can consider Perl6::Str; and for the ideal simpler Unicode handling scenario you described, there's no better prospect than Perl 6 but it's not yet ready for most users and use cases.

        Ah, I see! Yes, I was thinking that he suggested I look into Perl 6. I didn't realize Perl6::Str was actually a Perl 5 module. Thanks for the correction of my assumption there!

        This is perhaps veering off-topic, but since you mention "[getting] stuff done" (Perl 5) vs. "[having] fun figuring out how it works and contributing to the [...] effort by fixing and working around bugs and speed problems etc." (Perl 6) -- just how does Perl 6 really compare to Perl 5, then, how ready for production use is it? I keep getting conflicting information; on one hand there's advocates who will tell you that Perl 6 is Right Here to use, Right Now, while on the other you've got chronic naysayers making all sorts of disparaging statements.

        I reckon the truth is somewhere in between, but-- where? Performance problems aren't a big deal for me since I really only deal in small scripts; usually it's throw-away ones that only get used once, and if they get reused after all, it's usually only to massage data. Other than inefficiently-written code, speed is rarely an issue, and even inefficient code's often not such a big problem: I'd rather spend 10 seconds waiting for a script to finish if I'm only using it once than to spend 10 minutes optimizing it so it'll run in 1. ;)

        Bugs are a more serious matter. Of course they're inevitable in any language, and perl is gonna be no exception no matter whether we're talking about Perl 5 or Perl 6, but just how bug-free are the Perl 6 implentations (I gather there's several, right?) right now?

        for the ideal simpler Unicode handling scenario you described, there's no better prospect than Perl 6 but it's not yet ready for most users and use cases.

        What I'm taking home from this is "Perl 6 will be great once it works reliably, but it doesn't yet". Does that sound about right?

Re^3: length() miscounting UTF8 characters?
by farang (Hermit) on Apr 28, 2014 at 20:51 UTC

    Well, all perl builtins work at the codepoint level, including length. Depending on your definition of "character", that might or might not be what the OP wants.
    Sure, I'm just saying that bugs or unexpected results can occur if care is not taken. As amon pointed out, the same visual representation of a character with a diacritical might have either one or two codepoints.
    #!/usr/bin/env perl use v5.14; use warnings; use utf8; binmode STDOUT, 'utf8'; my $o_umlaut1 = "\x{F6}"; my $o_umlaut2 = "\x{6F}\x{308}"; my $string1 = "" . $o_umlaut1; my $string2 = "" . $o_umlaut2; say "length of $string1 is ", length($string1); say "length of $string2 is ", length($string2);
    __OUTPUT__
    length of  is 3
    length of ö is 4
    

    I'll play around with your module. Thai is somewhat unique in that the first combining character may be another alphabetic character, so counting extended graphemes does not necessarily give the correct count of alphabetic characters.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1084181]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (13)
As of 2014-07-28 19:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (207 votes), past polls