Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

Re: Modern best practices for multilingual regexp alphabetical character matching?

by ikegami (Pope)
on Jan 12, 2009 at 21:40 UTC ( #735814=note: print w/replies, xml ) Need Help??

in reply to Modern best practices for multilingual regexp alphabetical character matching?

The data I'm working with will all be UTF-8, if that makes a difference.

Make sure it's decoded using one or more of the following

use utf8; # Treat the source code as UTF-8 use open ':std', ':locale'; # Treat STD* as per locale use open ':std', ':encoding(UTF-8)'; # Treat STD* as UTF-8 use open IO => ':encoding(UTF-8)'; # Treat files as UTF-8 by default open(my $fh, '<:encoding(UTF-8)', $qfn) # Treat a file as UTF-8 utf8::decode(my $text = $encoded_text) # Treat a string as UTF-8 or die;

And make sure the string us stored internally as UTF-8.

utf8::upgrade($s); # Use UNICODE semantics

(No need to do use utf8; to use utf8:: functions. use utf8; means the source is in UTF-8.)

If you do those two things, regexp will use UNICODE semantics, so \w and character classes will match accented letters, etc.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://735814]
[marto]: heh, there's some truth to this
[marto]: youtube warning
Discipulus rh7 is converging towards win, while win converges towards linux..
[karlgoethebier]: marto: ...but you are not the interviewer, right?

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2017-12-14 11:10 GMT
Find Nodes?
    Voting Booth?
    What programming language do you hate the most?

    Results (389 votes). Check out past polls.