comment on

Here's a little script I cooked up not long ago to "deaccent" letters -- you need Perl version 5.8.0 or later to run it, and it assumes that your input text (from STDIN or file(s) named on the command line) is in utf-8:

#!/usr/bin/perl -CDS

use strict;
require 5.008;

my @charnames = grep /\tLATIN \S+ LETTER/, split( /^/, do 'unicore/Nam
+e.pl' );

my %accents;

for my $c ( split //, qq/AEIOUCNYaeioucny/ ) {
    my $case = ( $c eq lc $c ) ?  'SMALL' : 'CAPITAL';
    $accents{$c} =
          join( '', map { chr hex( substr $_, 0, 4 ) }
                grep /\tLATIN $case LETTER \U$c WITH/, @charnames );
}

# now use each element of %accents as a character class:

while (<>) {
    for my $c ( keys %accents ) {
        s/[$accents{$c}]/$c/g;
    }
    print;
}
[download]

If your original text is not utf8, well, you have to know what the encoding really is; then you can either find a way to convert to utf8 (e.g. there's an "iconv" tool on many systems, or you can use the Encode module in perl, which isn't that tough, really), OR you can hard-code all those conversions by hand instead of using the script shown above.

Based on one of your replies, you would be happy with converting the accented characters to symbolic entity references (á and so on). I think your hard-coded hash is as good a solution as any for that, so long as the encoding you used to write the the perl code matches the encoding of your text data.

In reply to Re^3: Removing Foreign Characters by graff
in thread Removing Foreign Characters by existem

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Pathologically Eclectic Rubbish Lister
	PerlMonks