|
Re^3: length() miscounting UTF8 characters?
by AppleFritter (Vicar) on Apr 28, 2014 at 19:47 UTC
|
Yes, "extended grapheme clusters" are what I'm apparently interested and what I'd ordinarily call "characters", rather than codepoints.
I've not looked at Perl 6 yet, but being able to work with Unicode data from a high-level perspective, without caring too much about implementation details such as the various representation layers (the encoding layer that take bytes to codepoints, and then the next one that takes codepoints to "extended grapheme clusters") would be a huge boon for many, including me.
| [reply] |
|
|
You seem to be thinking that Moritz is suggesting you use or look in to Perl 6. I'm pretty sure he is not.
Yes, Perl 6 does aim to enable working from the high-level perspective you describe.
No, it's not worth you looking in to Perl 6 if you just want to get stuff done and don't care to have fun figuring out how it works and contributing to the Perl 6 effort by fixing and working around bugs and speed problems etc.
Moritz mentioned Perl6::Str. This is a perl 5 module he wrote that must be used by perl 5 scripts. It is one of several perl (5) modules written by perl 5 programmers who chose to use the Perl6 namespace in CPAN to reflect the module being somehow related to Perl 6.
To quote the Perl6::Str module description:
Perl 5 offers string manipulation at the byte level (for non-upgraded strings) and at the codepoint level (for decoded strings). However it fails to provide string manipulation at the grapheme level, that is it has no easy way of treating a sequence of codepoints, in which all but the first are combining characters (like accents, for example) as one character.
Perl6::Str tries to solve this problem by introducing a string object with an API similar to that of Perl 6 (as far as possible), and emulating common operations such as substr, chomp and chop at the grapheme level. It also introduces builtin string methods found in Perl 6 such as samecase.
In summary, aiui: for production grade grapheme level handling, it's best to rely on perl 5, accepting all the intricacies and complications that inevitably arise; for some Perl 6 like features in perl 5 that attempt to hide some of that complexity (in trade for some loss of some magic and speed) you can consider Perl6::Str; and for the ideal simpler Unicode handling scenario you described, there's no better prospect than Perl 6 but it's not yet ready for most users and use cases.
| [reply] |
|
|
Ah, I see! Yes, I was thinking that he suggested I look into Perl 6. I didn't realize Perl6::Str was actually a Perl 5 module. Thanks for the correction of my assumption there!
This is perhaps veering off-topic, but since you mention "[getting] stuff done" (Perl 5) vs. "[having] fun figuring out how it works and contributing to the [...] effort by fixing and working around bugs and speed problems etc." (Perl 6) -- just how does Perl 6 really compare to Perl 5, then, how ready for production use is it? I keep getting conflicting information; on one hand there's advocates who will tell you that Perl 6 is Right Here to use, Right Now, while on the other you've got chronic naysayers making all sorts of disparaging statements.
I reckon the truth is somewhere in between, but-- where? Performance problems aren't a big deal for me since I really only deal in small scripts; usually it's throw-away ones that only get used once, and if they get reused after all, it's usually only to massage data. Other than inefficiently-written code, speed is rarely an issue, and even inefficient code's often not such a big problem: I'd rather spend 10 seconds waiting for a script to finish if I'm only using it once than to spend 10 minutes optimizing it so it'll run in 1. ;)
Bugs are a more serious matter. Of course they're inevitable in any language, and perl is gonna be no exception no matter whether we're talking about Perl 5 or Perl 6, but just how bug-free are the Perl 6 implentations (I gather there's several, right?) right now?
for the ideal simpler Unicode handling scenario you described, there's no better prospect than Perl 6 but it's not yet ready for most users and use cases.
What I'm taking home from this is "Perl 6 will be great once it works reliably, but it doesn't yet". Does that sound about right?
| [reply] |
|
|
|
|
|
|
|
|
Representing the written languages of the world on computers is complex. The Unicode Standard is complex. Programming Unicode is complex. There's a limit to how much of this complexity can be hidden from computer programmers.
If you want to understand Unicode better, and how to think correctly about programming Unicode, read Tom Christiansen's excellent Stack Overflow post here. If, after reading his well-known post, you find you need more of Tom's Perl Unicode wisdom, then come back to PerlMonks and read what tchrist has written about the topic here.
| [reply] |
|
|
OK, thanks again. Whoa, that's a lot of information to digest. I'll have to meditate on this, as it were.
| [reply] |
Re^3: length() miscounting UTF8 characters?
by farang (Chaplain) on Apr 28, 2014 at 20:51 UTC
|
Well, all perl builtins work at the codepoint level, including length. Depending on your definition of "character", that might or might not be what the OP wants.
Sure, I'm just saying that bugs or unexpected results can occur if
care is not taken. As amon pointed out, the same visual
representation of a character with a diacritical might have either
one or two codepoints.
#!/usr/bin/env perl
use v5.14;
use warnings;
use utf8;
binmode STDOUT, 'utf8';
my $o_umlaut1 = "\x{F6}";
my $o_umlaut2 = "\x{6F}\x{308}";
my $string1 = "æð" . $o_umlaut1;
my $string2 = "æð" . $o_umlaut2;
say "length of $string1 is ", length($string1);
say "length of $string2 is ", length($string2);
__OUTPUT__
length of æðö is 3
length of æðö is 4
I'll play around with your module. Thai is somewhat unique
in that the first combining character may be another alphabetic
character, so counting extended graphemes does not necessarily give
the correct count of alphabetic characters.
| [reply]
[d/l] |