Re^2: Replacing non ascii in string

[IanD]

in reply to Re: Replacing non ascii in string
in thread Replacing non ascii in string

Thanks, this does basically work but for some reason it is returning 3 x ' which I can't see why but it is working.

ie
$data_file =~ tr/‘’/'/;

returns
Australia'''s '''Powder Capital'''

Even though the input is "Australia’s ‘Powder Capital’"

[Athanasius]

I can’t reproduce this problem. Can you provide a complete but minimal script that demonstrates the behaviour you are seeing?

Please specify input and output precisely, and also show the output from perl -v.

[IanD]

I have solved by adding another line

$url = "http://www.flightcentre.com.au/static/SkiHolidays.xml";
$data_string = get($url);
$data_string =~ tr/‘’/'/;
$data_string =~ tr/’/'/;
$data_string =~ s/'''/'/g;

For some reason when I view the source of that xml I see the apostrophes that I show here. However to check it I was doing the following:

$file1 = "aafc.txt";

open (FILE,">$file1");
    print FILE $data_string;
close(FILE); 

Here I was seeing the string as
’Powder Capital’ instead of ‘Powder Capital’ as in the source of the xml. I don't understand why, clearly there has been some type of conversion of character types on reading the file. By making the substitution for ’ instead of ‘ it worked but then produced 3 ''' instead. So by doing that string replacement it worked. Seems bizarre to me but it works!

[Anonymous Monk]

Sounds like an encoding problem. Is your string utf-8? (Encode, $decoded_str = decode('utf-8', $str)) Did you use utf8 to get your literals parsed as such?

(You see, those fancy apostrophes are represented as three bytes. If Perl thinks we're still in ascii-land (binary-land), it sees the transliteration as tr/\xe2\x80\x99/'/ -- effectively changing any of those three bytes to an apostrophe.)

[IanD]

Bingo - that makes sense.

Sorted now, thanks for your assistance.