Re: keeping diacritical marks in a string

[graff]

Thanks for the update. I had no luck with the urls you posted, but I was able to go to the web page, put in a request that yielded accented characters in the output, and use the resulting url to push that request through LWP.

Since I got different content from what you were getting, your regex didn't really apply for my data (and I guess your regex isn't related to the problem anyway, since it has nothing to do with accented letters). Anyway, here's some code that demonstrates how the non-ascii content works:

#!/usr/bin/perl

use strict;
use LWP::UserAgent;
use HTTP::Request;

use Encode;  # you need this module

binmode STDOUT, ":utf8";

my $ua = LWP::UserAgent->new;

my $url = "some_url_that_works_for_you";
my $req = HTTP::Request->new( GET => $url );

my $res = $ua->request( $req );
$txt = decode( 'utf-8', $res->content );  # decode "external" utf8 to 
+"internal"

my @accented = ( $txt =~ /(\w*?[^[:ascii:]]\w*)/g );
if ( @accented ) {
    printf( "found %d words with non-ascii characters.\n", scalar @acc
+ented );

    my @alphanumerics = grep /^\w+$/, @accented;
    printf( "of those, %d words match ^\\w+\$:\n ", scalar @alphanumer
+ics );
    print join( "\n ", @alphanumerics ),"\n";

    my @diacritic_marks = grep /\p{NonspacingMark}/, @accented;
    printf( "and %d used separate diacritic marks:\n ", scalar @diacri
+tic_marks );
    print join( "\n ", @diacritic_marks ), "\n";
}

Having tried it myself, I learned that non-spacing diacritic marks (presented as separate characters, rather than being an intrinsic part of a letter -- e.g. the second character in "U+0061 U+02CA" for á, rather than U+00E1) all fall into the category of things that match "\w".

You might want to check out this little command-line tool I posted a while back -- it can really help with getting a handle on what kinds of unicode data you are really dealing with: tlu -- TransLiterate Unicode; check my home page for a few other unicode tools.

(UPDATE: Forgot to mention -- I also noticed that the source data from the web site tended to use both the single-character "accented_letter" and the two-character "letter accent_mark" for the same thing -- that is, their unicode usage is inconsistent, and somewhat non-standard.)