Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Modern best practices for multilingual regexp alphabetical character matching?

by ikegami (Patriarch)
on Jan 12, 2009 at 21:40 UTC ( [id://735814]=note: print w/replies, xml ) Need Help??


in reply to Modern best practices for multilingual regexp alphabetical character matching?

The data I'm working with will all be UTF-8, if that makes a difference.

Make sure it's decoded using one or more of the following

use utf8; # Treat the source code as UTF-8 use open ':std', ':locale'; # Treat STD* as per locale use open ':std', ':encoding(UTF-8)'; # Treat STD* as UTF-8 use open IO => ':encoding(UTF-8)'; # Treat files as UTF-8 by default open(my $fh, '<:encoding(UTF-8)', $qfn) # Treat a file as UTF-8 utf8::decode(my $text = $encoded_text) # Treat a string as UTF-8 or die;

And make sure the string us stored internally as UTF-8.

utf8::upgrade($s); # Use UNICODE semantics

(No need to do use utf8; to use utf8:: functions. use utf8; means the source is in UTF-8.)

If you do those two things, regexp will use UNICODE semantics, so \w and character classes will match accented letters, etc.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://735814]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (2)
As of 2024-07-23 06:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?
    erzuuli‥ 🛈The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.