Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Regex Matching Unicode and Regex Classes

by McA (Priest)
on Nov 02, 2011 at 13:53 UTC ( #935400=perlquestion: print w/replies, xml ) Need Help??
McA has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, I have the following little script demonstrating a case (case 1 in the output below), which I can't explain to myself. So I hoped, someone can explain it to me or give the right hints.
#!/usr/bin/perl -CO use strict; use warnings; use Encode; use utf8; my $a = 'ä'; print "UTF8-Flag: ", utf8::is_utf8($a) ? "Yes" : "No"; print " matches word: ", $a =~ /\w/ ? "Yes\n" : "No\n"; my $b = encode("ISO-8859-1", $a); print "UTF8-Flag: ", utf8::is_utf8($b) ? "Yes" : "No"; print " matches word: ", $b =~ /\w/ ? "Yes\n" : "No\n"; use locale; $a = 'ä'; print "UTF8-Flag: ", utf8::is_utf8($a) ? "Yes" : "No"; print " matches word: ", $a =~ /\w/ ? "Yes\n" : "No\n"; $b = encode("ISO-8859-1", $a); print "UTF8-Flag: ", utf8::is_utf8($b) ? "Yes" : "No"; print " matches word: ", $b =~ /\w/ ? "Yes" : "No"; print "\n";
The output on a linux box with locale de_DE.UTF-8 and perl source code encoded in UTF-8 is:
UTF8-Flag: Yes matches word: Yes UTF8-Flag: No matches word: No UTF8-Flag: Yes matches word: No UTF8-Flag: No matches word: No
It's the very first case I can't explain to me. Why is an unicode-flagged 'ä' matched against words when locale is not set explicitly?

Thanks in advance

Replies are listed 'Best First'.
Re: Regex Matching Unicode and Regex Classes
by choroba (Chancellor) on Nov 02, 2011 at 14:05 UTC
    Two days ago, I was having the same issues. My testing script is a bit more elaborate, but confirms your problem: \w does not work under use locale, whatever the locale is. If you need locale, use posix classes or unicode character properties (provided your locale is unicode based).
    Discussion on the details can be found in perlunicode and perllocale, but the relevant paragraphs are quite different in each version of Perl.
    #!/usr/bin/perl use warnings; use strict; my @p_locale = qw/C cs_CZ.UTF8 en_US.UTF8/; my @locale = ('no locale', 'use locale'); my @posix = (q(), 'use POSIX qw/locale_h/; setlocale LC_ALL,"C"', 'use POSIX qw/locale_h/; setlocale LC_ALL,"cs_CZ.UTF8"', ); for my $p_locale (@p_locale) { for my $locale (@locale) { for my $posix (@posix) { print "$p_locale $locale $posix\n"; open my $OUT, '>', 'l.perl' or die "$!"; print {$OUT} << " OUT"; $locale; $posix; my \@chars = qw/283 269 345 225 32 98 99 48 49 50 51 52 32 353 253 382 237/; my \$string = join q[], map chr, \@chars; binmode STDOUT, ':utf8'; my \@regex = (qr/(\\w+)/, qr/([[:alnum:]]+)/, qr/(\\p{W +ord}+)/); for my \$regex (\@regex) { print "\t\$1" while \$string =~ /\$regex/g; print "\n"; } print sort qw/ci ch/; print "\n"; OUT close $OUT; $ENV{LC_ALL} = $p_locale; system 'perl', 'l.perl'; } } } unlink 'l.perl';
      Hi choroba,
      thank you for your answer. I just checked it on my platform.

      By the way: A program generating code, mutating and running it, is called a virus. ;-)

      Best regards

      Tested on several versions of Perl: In 5.10.1 and 5.12.1, \w does not work. In 5.14.1, \w and [[:alnum:]] do not work. Should I report a bug?
Re: Regex Matching Unicode and Regex Classes
by moritz (Cardinal) on Nov 02, 2011 at 14:08 UTC

    The default Unicode semantics just check the Unicode properties of a codepoint. "ä" is U+00E4 LATIN SMALL LETTER A WITH DIAERESIS and classified as a letter, so \w matches it.

      Hi Moritz,

      but what is then the difference to the third case? Is the "default Unicode semantic" changed to something different when local is enabled?

      Why is "U+00E4 LATIN SMALL LETTER A WITH DIAERESIS" under locale something different than a letter which is part of a word?

      Best regards

        Short answer: because Unicode and locales don't mix.

        Long answer: Perl's support for locales comes from a time before the whole encoding/decoding business and Unicode support. So if locales are active, the locale-sensitive parts expect to act on bytes, not on decoded strings.

        Since the locale is not ISO-8859-1 but UTF-8, encoding to Latin-1 doesn't fix it for you.

        If anything, you'd need to encode to UTF-8 to see the \w matching ä, but even then I don't see it matching. So either my understanding of locales is very wrong, or perl is broken (or a mixture thereof).


      thanky ou for your answers.
      Have a nice day.

      Best regards

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://935400]
Approved by moritz
Front-paged by GrandFather
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (3)
As of 2017-08-20 00:51 GMT
Find Nodes?
    Voting Booth?
    Who is your favorite scientist and why?

    Results (313 votes). Check out past polls.