Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^3: length() miscounting UTF8 characters?

by AppleFritter (Curate)
on Apr 28, 2014 at 19:47 UTC ( #1084189=note: print w/ replies, xml ) Need Help??


in reply to Re^2: length() miscounting UTF8 characters?
in thread length() miscounting UTF8 characters?

Yes, "extended grapheme clusters" are what I'm apparently interested and what I'd ordinarily call "characters", rather than codepoints.

I've not looked at Perl 6 yet, but being able to work with Unicode data from a high-level perspective, without caring too much about implementation details such as the various representation layers (the encoding layer that take bytes to codepoints, and then the next one that takes codepoints to "extended grapheme clusters") would be a huge boon for many, including me.


Comment on Re^3: length() miscounting UTF8 characters?
Re^4: length() miscounting UTF8 characters?
by Jim (Curate) on Apr 28, 2014 at 21:55 UTC

    Representing the written languages of the world on computers is complex. The Unicode Standard is complex. Programming Unicode is complex. There's a limit to how much of this complexity can be hidden from computer programmers.

    If you want to understand Unicode better, and how to think correctly about programming Unicode, read Tom Christiansen's excellent Stack Overflow post here. If, after reading his well-known post, you find you need more of Tom's Perl Unicode wisdom, then come back to PerlMonks and read what tchrist has written about the topic here.

      OK, thanks again. Whoa, that's a lot of information to digest. I'll have to meditate on this, as it were.
Re^4: length() miscounting UTF8 characters?
by raiph (Hermit) on May 01, 2014 at 07:49 UTC
    You seem to be thinking that Moritz is suggesting you use or look in to Perl 6. I'm pretty sure he is not.

    Yes, Perl 6 does aim to enable working from the high-level perspective you describe.

    No, it's not worth you looking in to Perl 6 if you just want to get stuff done and don't care to have fun figuring out how it works and contributing to the Perl 6 effort by fixing and working around bugs and speed problems etc.

    Moritz mentioned Perl6::Str. This is a perl 5 module he wrote that must be used by perl 5 scripts. It is one of several perl (5) modules written by perl 5 programmers who chose to use the Perl6 namespace in CPAN to reflect the module being somehow related to Perl 6.

    To quote the Perl6::Str module description:

    Perl 5 offers string manipulation at the byte level (for non-upgraded strings) and at the codepoint level (for decoded strings). However it fails to provide string manipulation at the grapheme level, that is it has no easy way of treating a sequence of codepoints, in which all but the first are combining characters (like accents, for example) as one character.

    Perl6::Str tries to solve this problem by introducing a string object with an API similar to that of Perl 6 (as far as possible), and emulating common operations such as substr, chomp and chop at the grapheme level. It also introduces builtin string methods found in Perl 6 such as samecase.

    In summary, aiui: for production grade grapheme level handling, it's best to rely on perl 5, accepting all the intricacies and complications that inevitably arise; for some Perl 6 like features in perl 5 that attempt to hide some of that complexity (in trade for some loss of some magic and speed) you can consider Perl6::Str; and for the ideal simpler Unicode handling scenario you described, there's no better prospect than Perl 6 but it's not yet ready for most users and use cases.

      Ah, I see! Yes, I was thinking that he suggested I look into Perl 6. I didn't realize Perl6::Str was actually a Perl 5 module. Thanks for the correction of my assumption there!

      This is perhaps veering off-topic, but since you mention "[getting] stuff done" (Perl 5) vs. "[having] fun figuring out how it works and contributing to the [...] effort by fixing and working around bugs and speed problems etc." (Perl 6) -- just how does Perl 6 really compare to Perl 5, then, how ready for production use is it? I keep getting conflicting information; on one hand there's advocates who will tell you that Perl 6 is Right Here to use, Right Now, while on the other you've got chronic naysayers making all sorts of disparaging statements.

      I reckon the truth is somewhere in between, but-- where? Performance problems aren't a big deal for me since I really only deal in small scripts; usually it's throw-away ones that only get used once, and if they get reused after all, it's usually only to massage data. Other than inefficiently-written code, speed is rarely an issue, and even inefficient code's often not such a big problem: I'd rather spend 10 seconds waiting for a script to finish if I'm only using it once than to spend 10 minutes optimizing it so it'll run in 1. ;)

      Bugs are a more serious matter. Of course they're inevitable in any language, and perl is gonna be no exception no matter whether we're talking about Perl 5 or Perl 6, but just how bug-free are the Perl 6 implentations (I gather there's several, right?) right now?

      for the ideal simpler Unicode handling scenario you described, there's no better prospect than Perl 6 but it's not yet ready for most users and use cases.

      What I'm taking home from this is "Perl 6 will be great once it works reliably, but it doesn't yet". Does that sound about right?

        This post is about the immature Perl 6, not the rock solid Perl 5

        Performance problems aren't a big deal for me

        What if something that took a minute in P5 took a week in P6? OK, so it's no longer that bad, but Rakudo's speed can still be a problem for folk who don't think they care about speed. (This page of benchmarks appropriately takes forever to load.)

        how bug-free are the Perl 6 implentations (I gather there's several, right?)

        P6 is defined by a spec, so there have been multple implementations, and others could one day emerge, but realistically Rakudo is the only serious game in town.1

        I don't think I can usefully answer your "bug-free" question. So I'll be conservative and go with "expect lots of bugs".

        There are other serious issues, eg patchy documentation, very few modules, and no way to call Perl 5 code. Maybe one day...


        If the above hasn't put you off P6, use an IRC client or click this link to visit the freenode IRC channel #perl6 (log) between 6am and 6pm EST (most core devs are European), ask lots of (respectful!) questions, and type code in to the on-channel evalbots. If it isn't fun, forget P6. If it is, then maybe it'll also become useful to you. Either way, please post back here at the monastery about your take away. :)


        1 Rakudo relies on the underlying NQP compiler toolchain, so you may hear about that too. And NQP in turn runs on multiple backends including Parrot, JVM and the new MoarVM. So you may hear about those too. But these are all about Rakudo.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1084189]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (10)
As of 2014-10-20 23:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (92 votes), past polls