Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Answer: How do I normalize (e.g. strip) diacritical märks from a Unicode string?

by moritz (Cardinal)
on Apr 17, 2010 at 07:36 UTC ( #835241=categorized answer: print w/replies, xml ) Need Help??

Q&A > strings > How do I normalize (e.g. strip) diacritical märks from a Unicode string? - Answer contributed by moritz

The trick is to split the letters with diacritical marks into the base letter and the mark, which Unicode::Normalize does with the NFD function. Then the regex /\pM/ identifies marking characters (see perlunicode).
use strict; use warnings; use utf8; use Unicode::Normalize; my $s = "söme stüff\n"; $s = NFD($s); $s =~ s/\pM//g; print $s;

Depending on the application, the NFKD might or might not be more appropriate than NFD.

The code snippet above removes all marking characters, not just diacritical marks. You can change that by removing only \x{308}. The following code strips the diacritical mark, but leaves the accents:

use strict; use warnings; use utf8; use Unicode::Normalize; binmode STDOUT, ':utf8'; my $s = "söme stüff with áccènts\n"; $s = NFD($s); $s =~ s/\x{308}//g; $s = NFC($s); print $s;

Log In?
Username:
Password:

What's my password?
Create A New User
Chatterbox?
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2016-12-03 12:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:













    Results (55 votes). Check out past polls.