Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^6: Fatal code point 0xFFFFFFFFFFFFFFFF

by dave_the_m (Monsignor)
on Sep 09, 2018 at 15:47 UTC ( #1221984=note: print w/replies, xml ) Need Help??


in reply to Re^5: Fatal code point 0xFFFFFFFFFFFFFFFF
in thread Fatal code point 0xFFFFFFFFFFFFFFFF

I can't believe Perl is gonna die just because of "illegal" unicodes in the regex engine
In the past the perl interpreter would often crash or leave itself open to security holes when fed badly-formed utf8. Now it more often detects such cases early and refuses to continue (with just warning for a grace period to allow people to fix up their code).

There are 7,389 lines in the main perl interpreter C source which have the the text 'utf' in them. If you want to volunteer to do a thorough audit of every single one of those lines to look for, e.g. negative wrap-around, performance degradation and security issues, and want to write comprehensive tests for every case, for both 32-bit and 64-bit platforms, then its possible that the decision to ban code-points above 2^(N-1) might be rescinded.

But there really is no point. If you're feeding codepoints > 0x7fffffffffffffff into perl, something fairly major has gone wrong, and quietly ignoring those rogue codepoints is just masking the bigger issue.

Dave.

  • Comment on Re^6: Fatal code point 0xFFFFFFFFFFFFFFFF

Replies are listed 'Best First'.
Re^7: Fatal code point 0xFFFFFFFFFFFFFFFF
by Anonymous Monk on Sep 09, 2018 at 23:00 UTC
    If you're feeding codepoints > 0x7fffffffffffffff into perl, something fairly major has gone wrong, and quietly ignoring those rogue codepoints is just masking the bigger issue.

    This excellent sentence got me pointed in the right direction! Thank you. Here's what the regex engine was freaking out about:

    Encode::decode(utf8 => $binary_data) # from binmode filehandle
    Matching that data triggers the fatal code point error, but only if the regex fails to match a string (numbers seem to match or fail quietly). Different versions of perl give different results.

    Perl 5.26.2:

    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(f)/i;'
    F
    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fu)/i;'
    fu
    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fub)/i;'
    Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA8 at -e line 1.

    Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x1C9140 at -e line 1.

    Operation "pattern match (m//)" returns its argument for non-Unicode code point 0xE6BAAA at -e line 1.

    Use of code point 0xFFFFFFFFFFFFFFFF is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF. This will be fatal in Perl 5.28 in pattern match (m//) at -e line 1.

    Use of code point 0xFFFFFFFFFFFFFFFF is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF. This will be fatal in Perl 5.28 at -e line 1.

    Operation "pattern match (m//)" returns its argument for non-Unicode code point 0xFFFFFFFFFFFFFFFF at -e line 1.


    Perl 5.28:
    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(f)/i;'
    f
    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fu)/i;'
    Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA9 at -e line 1.

    FU


    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fub)/i;'
    Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA9 at -e line 1.

    Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x1CB760 at -e line 1.

    Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x18B14C at -e line 1.

    Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x7FFFFFFFFFFFFFFF at -e line 1.


    Numbers:
    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(1)/i;'
    1
    perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(666)/i;'
      It's not clear to me what point you're trying to make with the code examples.

      Dave.

        It's not clear to me what point you're trying to make with the code examples.

        A fatal code point Dave! :-) Next time someone accidentally tries to decode a binary file with Encode and Perl complains about fatal code points they may find this node and save the day it took me to figure out what was wrong. Then the fix is easy.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1221984]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (5)
As of 2019-06-17 22:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Is there a future for codeless software?



    Results (80 votes). Check out past polls.

    Notices?
    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!