Excluding non ascii characters

[millsperl]

Need to exclude non ascii characters, i.e. greater than 7F(hex). Currently the perl script excludes a couple of characters, such as null. This is done by declaring the character to be excluded, i.e. $remove_char = chr(hex("00")); Then each line in the file is read and the null replaced by space, i.e. $myline =~ s/$remove_char/ /g; How should I code the declaration and statement that does the substitution?

[jwkrahn]

Perhaps you want something like this:

$myline =~ s/[^[:ascii:]]/ /g;

[linuxer]

What about a character range? And if you want to delete those characters, you also can use tr///.

$myline =~ tr/\x80-\xff//d;

or if you want to stick to s///.

$myline =~ s/[\x80-\xff]/ /g;

[ikegami]

That will only remove the non-ASCII characters that are in iso-latin-1, a very small portion of non-ASCII characters. Fix:

# Replace with space
$myline =~ s/[^\x00-\x7F]/ /g;

# Delete character
$myline =~ tr/\x00-\x7F//cd;
$myline =~ s/[^\x00-\x7F]//g;

[ikegami]

Here's a solution that works with arbitrary encodings rather than just ASCII:

use Encode qw( decode encode );

my $input    = '...';
my $enc      = 'US-ASCII';
my $spc      = encode($enc, ' ');
my $filtered = decode($enc, encode($enc, $s, sub { $spc }));

Encode

[ikegami]

You might be interested in Text::Unidecode

[Anonymous Monk]

The author hasn't updated it in over 10 years, and refuses to respond to bug reports, patches, or even accepting comaintainers.