Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Re: Comparing strings at grapheme level

by halley (Prior)
on Jan 09, 2008 at 14:51 UTC ( #661367=note: print w/ replies, xml ) Need Help??


in reply to Comparing strings at grapheme level

I just did a google search and it looks like this is fairly authoritative, but still not quite helpful in your situation.

  • http://unicode.org/reports/tr15/#Stability_of_Normalized_Forms
  • http://unicode.org/reports/tr15/#Canonical_Equivalence

    It would seem that if the normalization functions worked to that spec, then they would always produce the same output order. However, in my limited experience I've never seen any real-world cases where you'd want or need multiple COMBINING characters... not saying it doesn't exist but the Unicode docs don't obviously mention it specifically.

    I would say that if you're straying outside the defined normalization routines, that you gotta do something stable and predictable, like sort your multiple COMBINING elements into numerical codepoint order. (Order of these should never matter for the grapheme, right?)

    Update: http://useless-factor.blogspot.com/2007/07/unicode-implementers-guide-part-2.html This blog says ordering does matter, and that they should be sorted by class of combining element. The ordering of combining elements in the same class may or may not need to be maintained, which is the issue you're running into. Pretty annoying if there's not a standard library implementation of all this.

    My perlish hat says that to compare two graphemes, you'd have to do a hash/set of included characters, then compare the hashes for equivalence. If those match, and the order of combining classes is the same, it's the same grapheme.

    --
    [ e d @ h a l l e y . c c ]


  • Comment on Re: Comparing strings at grapheme level
    Re^2: Comparing strings at grapheme level
    by moritz (Cardinal) on Jan 10, 2008 at 10:02 UTC
      Thanks for your helpful post.

      At the moment I have no real Unicode problems (but I rather try to understand the concepts). I'll keep that in mind when I start do some real work with Unicode.

    Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Node Status?
    node history
    Node Type: note [id://661367]
    help
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others imbibing at the Monastery: (13)
    As of 2014-07-10 11:38 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      When choosing user names for websites, I prefer to use:








      Results (207 votes), past polls