Re^4: Fatal code point 0xFFFFFFFFFFFFFFFF

in reply to Re^3: Fatal code point 0xFFFFFFFFFFFFFFFF
in thread Fatal code point 0xFFFFFFFFFFFFFFFF

to suppress unicode's stupid errors.

They are not stupid errors. The errors (in fact warnings - they only become errors from 5.28 onwards) are telling you that you are trying to feed illegal data into the regex engine. It's illegal because the octets you told decode() to interpret as utf8 aren't in fact valid utf8.

How you want to handle this corrupt data is of course your choice depending on what is best for your circumstances. You might want to make decode croak if fed a bad file with FB_CROAK, then go back and delete / fixup any bad files. Or depending on the nature of the corruption of the files, you might like to use FB_DEFAULT to just replace the corrupt bits with REPLACEMENT CHARACTER. You would only want to use FB_QUIET if you don't mind decode() stopping at the first bad part of the file and ignoring the rest of its contents.

Dave.

Comment on Re^4: Fatal code point 0xFFFFFFFFFFFFFFFF

Replies are listed 'Best First'.
Re^5: Fatal code point 0xFFFFFFFFFFFFFFFF by Anonymous Monk on Sep 09, 2018 at 15:22 UTC
Thanks again Dave. I can't believe Perl is gonna die just because of "illegal" unicodes in the regex engine. PERL HAS A LICENSE TO ILL! Why don't they just warn and continue? Or do something mellow like ignore illegal code points, or just change uncool values to the max (and give weenies strict for unicode if they require torture)? Why be Unicode Security Guards? All these negative vibes while parsing something as important as 21st century text can't possibly be right for Perl: deprecated permissible fatal will not be allowed it is too large break the perl causing it to hang problem areas breakages warning upper limit out of range was deprecated a fatal error Perl 5.28 That's from one error message. I don't know who wrote that but they do not sound like a happy camper! Let's send them some love so they can transform all that dark energy into something wonderful, like this: `Use of code point 0xFFFFFFFFFFFFFFFF detected; but the permissible max is 0x7FFFFFFFFFFFFFFF. Perl can handle this at X.pm line 1685` `(D dude!) You used a code point that is way too large. Unicode only allows code points up to 0x10FFFF, but of course Perl allows much larger ones. However, the perl interpreter can handle anything. So your error has been automagically corrected! If your code is to run on various platforms, keep in mind that the upper limit depends on the platform. It is much larger on 64-bit word sizes than 32-bit ones.` `The use of out of range code points was acceptable in Perl 5.24, and it will be totally cool, but still pointless, in Perl 5.28.`	[reply]
Re^6: Fatal code point 0xFFFFFFFFFFFFFFFF by dave_the_m (Monsignor) on Sep 09, 2018 at 15:47 UTC
I can't believe Perl is gonna die just because of "illegal" unicodes in the regex engine In the past the perl interpreter would often crash or leave itself open to security holes when fed badly-formed utf8. Now it more often detects such cases early and refuses to continue (with just warning for a grace period to allow people to fix up their code). There are 7,389 lines in the main perl interpreter C source which have the the text 'utf' in them. If you want to volunteer to do a thorough audit of every single one of those lines to look for, e.g. negative wrap-around, performance degradation and security issues, and want to write comprehensive tests for every case, for both 32-bit and 64-bit platforms, then its possible that the decision to ban code-points above 2^(N-1) might be rescinded. But there really is no point. If you're feeding codepoints > 0x7fffffffffffffff into perl, something fairly major has gone wrong, and quietly ignoring those rogue codepoints is just masking the bigger issue. Dave.	[reply]
Re^7: Fatal code point 0xFFFFFFFFFFFFFFFF by Anonymous Monk on Sep 09, 2018 at 23:00 UTC
*If you're feeding codepoints > 0x7fffffffffffffff into perl, something fairly major has gone wrong, and quietly ignoring those rogue codepoints is just masking the bigger issue.* This excellent sentence got me pointed in the right direction! Thank you. Here's what the regex engine was freaking out about: `Encode::decode(utf8 => $binary_data) # from binmode filehandle` [download] Matching that data triggers the fatal code point error, but only if the regex fails to match a string (numbers seem to match or fail quietly). Different versions of perl give different results. Perl 5.26.2: `perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(f)/i;'` [download] F `perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fu)/i;'` [download] fu `perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fub)/i;'` [download] Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA8 at -e line 1. Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x1C9140 at -e line 1. Operation "pattern match (m//)" returns its argument for non-Unicode code point 0xE6BAAA at -e line 1. Use of code point 0xFFFFFFFFFFFFFFFF is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF. This will be fatal in Perl 5.28 in pattern match (m//) at -e line 1. Use of code point 0xFFFFFFFFFFFFFFFF is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF. This will be fatal in Perl 5.28 at -e line 1. Operation "pattern match (m//)" returns its argument for non-Unicode code point 0xFFFFFFFFFFFFFFFF at -e line 1. Perl 5.28: perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(f)/i;' [download] f perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fu)/i;' [download] Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA9 at -e line 1. FU perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(fub)/i;' [download] Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA9 at -e line 1. Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x1CB760 at -e line 1. Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x18B14C at -e line 1. Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x7FFFFFFFFFFFFFFF at -e line 1. Numbers: perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(1)/i;' [download] 1 perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde +f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data) +; print $1 if $data =~ /(666)/i;' [download]	[reply] [d/l] [select]
Re^8: Fatal code point 0xFFFFFFFFFFFFFFFF by dave_the_m (Monsignor) on Sep 10, 2018 at 07:14 UTC
Re^9: Fatal code point 0xFFFFFFFFFFFFFFFF by Anonymous Monk on Sep 10, 2018 at 07:34 UTC
Some notes below your chosen depth have not been shown here

In Section Seekers of Perl Wisdom