Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re: Modern best practices for multilingual regexp alphabetical character matching?

by ikegami (Pope)
on Jan 12, 2009 at 21:40 UTC ( #735814=note: print w/ replies, xml ) Need Help??


in reply to Modern best practices for multilingual regexp alphabetical character matching?

The data I'm working with will all be UTF-8, if that makes a difference.

Make sure it's decoded using one or more of the following

use utf8; # Treat the source code as UTF-8 use open ':std', ':locale'; # Treat STD* as per locale use open ':std', ':encoding(UTF-8)'; # Treat STD* as UTF-8 use open IO => ':encoding(UTF-8)'; # Treat files as UTF-8 by default open(my $fh, '<:encoding(UTF-8)', $qfn) # Treat a file as UTF-8 utf8::decode(my $text = $encoded_text) # Treat a string as UTF-8 or die;

And make sure the string us stored internally as UTF-8.

utf8::upgrade($s); # Use UNICODE semantics

(No need to do use utf8; to use utf8:: functions. use utf8; means the source is in UTF-8.)

If you do those two things, regexp will use UNICODE semantics, so \w and character classes will match accented letters, etc.


Comment on Re: Modern best practices for multilingual regexp alphabetical character matching?
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://735814]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2014-11-23 17:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My preferred Perl binaries come from:














    Results (134 votes), past polls