Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
Server Error (Error ID 1782693b65254)

An error has occurred. The site administrators have been notified of the problem and will likely soon fix it. We thank you, for you're patients.

( #835238=categorized question: print w/replies, xml ) Need Help??
Contributed by on Apr 17, 2010 at 07:24 UTC
Q&A  > strings


How do I normalize a Unicode string, folding or removing all diacritical marks and accents? I wish to do this in preparation for saving words for later searches.

Answer: How do I normalize (e.g. strip) diacritical märks from a Unicode string?
contributed by brycen

Unicode defines a variety of normalization forms (see

I prefer normalization form NFKD, as it translates more ligatures (though not all, for example the ligature Œ).

First decompose composite characters into their component parts (e.g. letters and diacritical marks), then strip out the marks.

$str = Unicode::Normalize::NFKD($str); $str =~ s/\p{NonspacingMark}//g;
Or with a full example:
## Demonstrate stripping of diacritical marks from Unicode strings ## April 2010, Bryce Nesbitt, Berkeley Electronic Press ## See also ## See also ## Keywords: perl, diacritic, diacritical ## accent, iso-8859-1, normalization. use utf8; # Tell perl source code is utf-8 use 5.10.0; use Unicode::Normalize; # Sample: "latin small letter e with circumflex and tilde" &#7877; # "latin small ligature ff" (will be expanded) # "latin small ligature oe" (won't be expanded) $str = shift || "\x{1ec5} märks \x{fb00} \x{153}"; say "Input: ".debug_chatty_string($str); # Decompose into letter and combining marks, in "Kompatibility" mode $str = NFKD($str); say "NFKD : ".debug_chatty_string($str); # Remove combining marks $str =~ s/\p{NonspacingMark}//g; $str = lc($str); say "Out : ".debug_chatty_string($str); sub debug_chatty_string { my $outstring; # Use shift below, so utf-8 flag is preserved. # Else you might have to fiddle with Encode::_utf8_on() foreach $char (split //,shift) { my $ord = ord($char); if(($ord >= 32 && $ord < 127) || $ord == 10) { $outstring .= $char; } else { $outstring .= "<0x".sprintf("%x",$ord).">"; } } return $outstring; }
Example run:
Input: <0x1ec5> m<0xe4>rks <0xfb00> <0x153> NFKD : e<0x302><0x303> ma<0x308>rks ff <0x153> Out : e marks ff <0x153>

Update: I really do mean normalization. ASCIIfying (e.g. encoding) would destroy non-latin text. Normalization preserves Greek, Hebrew, etc.

I am supporting clients in various languages who want the fuzzy matching that stripping diacriticals provides. It might make for the occasional confusion between German bears and bars... but that's much better than missing out on all the potential correct matches. For example in Hebrew vowels are not normally written except for children. Stripping the vowel and pronunciation diacriticals out lets you compare the text as an adult searcher will likely enter it.

Answer: How do I normalize (e.g. strip) diacritical märks from a Unicode string?
contributed by moritz

The trick is to split the letters with diacritical marks into the base letter and the mark, which Unicode::Normalize does with the NFD function. Then the regex /\pM/ identifies marking characters (see perlunicode).

use strict; use warnings; use utf8; use Unicode::Normalize; my $s = "söme stüff\n"; $s = NFD($s); $s =~ s/\pM//g; print $s;

Depending on the application, the NFKD might or might not be more appropriate than NFD.

The code snippet above removes all marking characters, not just diacritical marks. You can change that by removing only \x{308}. The following code strips the diacritical mark, but leaves the accents:

use strict; use warnings; use utf8; use Unicode::Normalize; binmode STDOUT, ':utf8'; my $s = "söme stüff with áccènts\n"; $s = NFD($s); $s =~ s/\x{308}//g; $s = NFC($s); print $s;
Answer: How do I normalize (e.g. strip) diacritical märks from a Unicode string?
contributed by ikegami

If by "normalize" you mean ASCIIfying text, this can be done using Text::Unidecode.

Please (register and) log in if you wish to add an answer

  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?

    What's my password?
    Create A New User
    and all is quiet...

    How do I use this? | Other CB clients
    Other Users?
    Others surveying the Monastery: (3)
    As of 2016-12-10 11:43 GMT
    Find Nodes?
      Voting Booth?
      On a regular basis, I'm most likely to spy upon:

      Results (162 votes). Check out past polls.