Replacing non ascii in string

[IanD]

OK I have searched and searched here and can't find a solution for this.

I have a string that contains copied and pasted text from word or some mac thing and it contains non ascii apostrophies and quotes.

eg (all in one long string):

Australia’s ‘Powder Capital’
and  ... xxx said “This is a fantastic start to the season”

I want to replace all the ‘ and ’ with the ascii ' and the “ and ” with the ascii "

I can get rid of them with this:
$data_string =~s/[^[:ascii:]]//g;

But I want to replace them, not remove them and can't for the life of me work out the right regex to do this.

Also what reference do you use for the lookup. I have been using this:
http://www.ascii.cl/htmlcodes.htm

Which could be my problem. As down the track I am sure I will want to replace things like ä with a etc as well.

Thanks.

[Kenosis]

Perhaps Text::Unidecode would be helpful (please forgive the lack of complete code formatting, as doing so eliminates displaying the characters the module decodes):

use strict;
use warnings;
use utf8;
use Text::Unidecode;

my $string = q/‘ and ’ “ and ” and ä/;

print unidecode($string);

Output:

' and ' " and " and a

[Athanasius]

Hello IanD, and welcome to the Monastery!

Try this:

14:33 >perl -wE "my $s = 'Australia’s ‘Powder Capital’'; $s =~ tr/‘’/'
+/; say $s;"
Australia's 'Powder Capital'

14:33 >

See tr{}{} in Quote and Quote like Operators.

Update: Likewise,

14:42 >perl -wE "my $t = 'and ... xxx said “This is a fantastic start 
+to the season”'; $t =~ tr/“”/\"/; say $t;"
and ... xxx said "This is a fantastic start to the season"

14:43 >

Or combined into one:

14:46 >perl -wE "my $s = qq[Australia’s ‘Powder Capital’\nand ... xxx 
+said “This is a fantastic start to the season”]; $s =~ tr/‘’“”/''\"\"
+/; say $s;"
Australia's 'Powder Capital'
and ... xxx said "This is a fantastic start to the season"

14:49 >

Hope that helps,

[IanD]

Thanks, this does basically work but for some reason it is returning 3 x ' which I can't see why but it is working.

ie
$data_file =~ tr/‘’/'/;

returns
Australia'''s '''Powder Capital'''


Even though the input is "Australia’s ‘Powder Capital’"

[Athanasius]

I can’t reproduce this problem. Can you provide a complete but minimal script that demonstrates the behaviour you are seeing?

Please specify input and output precisely, and also show the output from perl -v.

[Anonymous Monk]

Sounds like an encoding problem. Is your string utf-8? (Encode, $decoded_str = decode('utf-8', $str)) Did you use utf8 to get your literals parsed as such?

(You see, those fancy apostrophes are represented as three bytes. If Perl thinks we're still in ascii-land (binary-land), it sees the transliteration as tr/\xe2\x80\x99/'/ -- effectively changing any of those three bytes to an apostrophe.)

[vinoth.ree]

This will remove the non ascii characters,also creates backup file in file.backup file.

You can do the job with a perl one liner:   perl -i.backup -pe 's/[[:^ascii:]]//g;' file

Also, $str =~ s/[^!-~\s]//g; In the above, !-~ is a range which matches all characters between ! and ~. The range is set between ! and ~ because these are the first and last characters in the ASCII table.This does not include whitespace, so i added \s also.

[Anonymous Monk]

As down the track I am sure I will want to replace things like ä with a etc as well.

That doesn't sound like a good idea. ä is a letter in some languages, not just "a with diacritic". Writing a name without the "diacritic" could be downright wrong. (Yes, sports broadcasts frequently do it that way, but that doesn't mean you should.) Do you really need to pare all of your text down to 7-bit ASCII?

[MidLifeXis]

While I agree with your general point, it is not an absolute. For some systems I interact with, yes. Things that only support 7-bit ASCII are still out there.

Update: added 'absolute' comment.

[naChoZ]

Hardly an elegant solution, but my little script will do exactly what you want.

#!/usr/bin/perl -n

#use strict;
#use warnings;
use charnames ();
use encoding "utf8";

$|++;

my $chars = {
    'HYPHEN'                      => '-',    #  \x{2010}
    'SOFT HYPHEN'                 => '-',    #­ \x{00AD}
    'MINUS SIGN'                  => '-',    #  \x{2212}
    'FIGURE DASH'                 => '-',    #  \x{2012}
    'ACUTE ACCENT'                => "'",    #  \x{00B4}
    'GRAVE ACCENT'                => "'",    #  \x{0060}
    'LEFT SINGLE QUOTATION MARK'  => "'",    #  \x{2018}
    'RIGHT SINGLE QUOTATION MARK' => "'",    #  \x{2019}
    'LEFT DOUBLE QUOTATION MARK'  => '"',    #  \x{201C}
    'RIGHT DOUBLE QUOTATION MARK' => '"',    #  \x{201D}
    'BOX DRAWINGS LIGHT VERTICAL' => '|',    #  \x{2502}
    'MULTIPLICATION SIGN'         => '*',    #  \x{00D7}
    'BACKSPACE'                   => '',     #  \x{0008}
    'DELETE'                      => '',     #  \x{0127}
};


# If the first character is an equal sign, skip it and
# display the identity of each remaining character.
#
if (/^=/) {
    chomp;
    for my $index ( 1 .. length($_) - 1 ) {

        my $char = substr( $_, $index++, 1 );

        print $char . " "
                    . ord( $char )
                    . " "
                    . sprintf( "\\%03o", ord($char) )
                    . " "
                    . sprintf( "\\x{%04X}", ord($char) )
                    . " = '"
                    . charnames::viacode( ord($char) )
                    . "'\n" ;

    }

} else {

    for my $cname ( keys %$chars ) {

        my $char = chr( charnames::vianame($cname) );
        s/$char/$chars->{$cname}/g;

    }

    print;

}

Run this and it sits there waiting for you to paste something to your terminal. Once it receives some input, it starts replacing characters by their unicode name with the values from the $chars hashref.

If you need to identify any unicode characters because you want to add to the hashref of replacement characters, press the = key first and then paste to your terminal.