If you're feeding codepoints > 0x7fffffffffffffff into perl, something fairly major has gone wrong, and quietly ignoring those rogue codepoints is just masking the bigger issue.
This excellent sentence got me pointed in the right direction! Thank you. Here's what the regex engine was freaking out about:
Encode::decode(utf8 => $binary_data) # from binmode filehandle
Matching that data triggers the fatal code point error, but only if the regex fails to match a string (numbers seem to match or fail quietly). Different versions of perl give different results.
Perl 5.26.2:
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(f)/i;'
F
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(fu)/i;'
fu
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(fub)/i;'
Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA8 at -e line 1.
Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x1C9140 at -e line 1.
Operation "pattern match (m//)" returns its argument for non-Unicode code point 0xE6BAAA at -e line 1.
Use of code point 0xFFFFFFFFFFFFFFFF is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF. This will be fatal in Perl 5.28 in pattern match (m//) at -e line 1.
Use of code point 0xFFFFFFFFFFFFFFFF is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF. This will be fatal in Perl 5.28 at -e line 1.
Operation "pattern match (m//)" returns its argument for non-Unicode code point 0xFFFFFFFFFFFFFFFF at -e line 1.
Perl 5.28:
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(f)/i;'
f
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(fu)/i;'
Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA9 at -e line 1.
FU
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(fub)/i;'
Operation "pattern match (m//)" returns its argument for UTF-16 surrogate U+DFA9 at -e line 1.
Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x1CB760 at -e line 1.
Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x18B14C at -e line 1.
Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x7FFFFFFFFFFFFFFF at -e line 1.
Numbers:
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(1)/i;'
1
perl -MEncode -le 'open my $fh, "<", $^X; binmode $fh; local $/ = unde
+f; my $data = <$fh>; close $fh; $data = Encode::decode(utf8 => $data)
+; print $1 if $data =~ /(666)/i;'
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
