"use encoding" behaviour change under Perl 5.10?

[gnosek]

Hello all,

I upgraded some of my systems to Perl 5.10 and have fought for several hours with a change that boils down to:

# Perl 5.8
perl -Mencoding=utf8 -le 'print ord chr 156'
156

# Perl 5.10
perl -Mencoding=utf8 -le 'print ord chr 156'
65533

I cannot easily remove the 'use encoding "utf8"' from my script as the pragma does "some magic" that prevents encoding-related disasters (double-encoded strings) further down the road.

This change means (I really hope someone will correct me) that I have to maintain my own copy of CGI::Util with "use bytes" inside "sub unescape". The offending statement is:

      $todecode =~ s/%(?:([0-9a-fA-F]{2})|u([0-9a-fA-F]{4}))/
        defined($1)? chr hex($1) : utf8_chr(hex($2))/ge;

Does anybody know the reason behind this change? Would that be a bug or a bugfix?

Best regards,
 Grzegorz Nosek

[graff]

I wish I could answer your question, but all I can do is point out that comparing 5.8.8 and 5.10 in terms of their respective "encoding.pm" and "utf8.pm" files doesn't seem to help. The only real diffs (apart from changes in white-space usage) involve the POD, and those diffs don't really shed any light on this issue.

In one respect, it may be that 5.10's behavior is "consistent" in a way that 5.8's behavior is not:

# in perl 5.8.8:
perl -Mencoding=utf8 -e '$a="\x51";$b="\xE1"; printf( "a=%x b=%x\n",or
+d($a),ord($b))'
a=51 b=fffd

perl -Mencoding=utf8 -e '$a=chr(hex("51"));$b=chr(hex("E1")); printf( 
+"a=%x b=%x\n",ord($a),ord($b))'
a=51 b=e1

In 5.10, those two commands both produce "b=fffd". But knowing that probably doesn't help either (sorry).

While it's probably true that maintaining your own local version of CGI::Util will be "easier", it might still make sense to consider whether there's a better way to handle this issue:

I cannot easily remove the 'use encoding "utf8"' from my script as the pragma does "some magic" that prevents encoding-related disasters (double-encoded strings) further down the road.

Figuring what that "magic" is and where it applies in your code, and then looking for better ways to achieve the same result, might be better in the long run. In particular, since "use encoding" is not scoped, replacing that solution with some other (scoped and/or focused) approach for your encoding-related disasters would seem prudent and worthwhile.

[gnosek]

Thanks for your example. I understand that "use encoding" applies to the program source (string literals etc.) but that it affects the chr function was quite a surprise to me.

As for replacing it with something more manageable, I certainly will, but I needed a workaround right now and could not afford to dig into the code at the moment.

[jdd]

From perldoc perlunicode:

The "chr()" and "ord()" functions work on characters, similar to "pack("W")" and "unpack("W")", not "pack("C")" and "unpack("C")". "pack("C")" and "unpack("C")" are methods for emulating byte-oriented "chr()" and "ord()" on Unicode strings. While these methods reveal the internal encoding of Unicode strings, that is not something one normally needs to care about at all.

Being a no-expert at all in that, just hope the following can help to give you or others a good direction:

perl -Mencoding=utf8 -le 'print unpack "C", chr 156'
156

perl -Mencoding=utf8 -M'Encode qw(from_to)' -le '$c = chr 156; from_to($c, "iso-8859-3", "utf-8"); print ord $c'
156

[jdd]

Although not answering to the question, I also fighted with CGI::Util and UTF8 decoding, and here is my fix. I ended up with replacing

utf8_chr(hex($2))

by

pack("U", hex($2))

which seems to me less hacking that the CGI::Util's private utf8_chr() routine. No need of use bytes; in my case.

[jdd]

Btw, more or less off-topic, CGI::Util can unescape unicode with the uXXXX convention but does not follow this convension when doing escaping -; i.e. I changed its encoding routine from

$toencode=~s/([^a-zA-Z0-9_.~-])/uc sprintf("%%%02x",ord($1))/eg;
to

$toencode =~ s/([^a-zA-Z0-9_.-])/(ord($1) > 255) ? sprintf("%%u%04X", ord($1)) : sprintf("%%%02X", ord($1))/eg;

[gnosek]

Thanks, it hasn't bitten me yet but I'll keep an eye on it. I have encountered problems in the other half of the expression.

A form field value of '<ę' (less than sign, small e with ogonek) got urlencoded as '%3C%C4%99' by the browser and decoded as '<\x{fffd}\x{fffd}' by 'chr(hex($1))'. What really drove me mad was that 'ę' itself ('%C4%99') was handled properly so I thought it was some HTML escaping misfeature that appeared between CGI versions.

[diotalevi]

You've got another problem. Your browser incorrectly passed the ę character as %C4%88 that is, as a string which happens to be UTF-8 but should have passed it as %u0119. You've got to *guess* during your URL parsing if someone has started passing you a byte-string which happens to look like a UTF-8 string and if so, reinterpret it as UTF-8 instead. If you're being fully careful, you should also be using the Content-Type of the HTTP header your server sends, the client sends, and the HTML content encoding you sent when generating your page.

Here's what I use.

sub uri_unescape {
    my $to_decode = shift;

    return if ! defined $to_decode;
    $to_decode =~ s/\+/ /g;

    # A "good enough" check to see if the browser encoded high bit
    # characters as UTF-8 by looking for something that resembles a
    # URL-encoded utf8 octet series
    #
    # utf8 encoding starts with 11xxxxxx in the first byte and
    # 10xxxxxx in the subsequent bytes
    # http://en.wikipedia.org/wiki/UTF-8#Description
    my $is_byte_encoded_utf8 = $to_decode =~ /%[c-fC-F][[:xdigit:]]%[8
+9abAB][[:xdigit:]]/;

    $to_decode =~ s/%([[:xdigit:]]{2})/chr hex "0x$1"/eg;

    # Decode the "UTF-8" if it looked like it was such. If I enter
    # this block, the string may now be UTF-8 encoded as a perl
    # string.
    if ($is_byte_encoded_utf8) {
        $to_decode = Encode::decode( 'utf8', $to_decode );
    }

    # Promote %uXXXX escapes up to characters. This is after the
    # Encode call so I don't inadvertently cause a promotion to utf8
    # and then ask Encode to do another promotion.
    $to_decode =~ s/%u([[:xdigit:]]{4})/chr hex "0x$1"/eg;

    return $to_decode;
}

[Anonymous Monk]

Both ?:)

perl -Mutf8 -le "print ord chr 156"

prints 156 for both 5.8.4 and 5.10.0