Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re^2: Fatal code point 0xFFFFFFFFFFFFFFFF

by dave_the_m (Monsignor)
on Sep 09, 2018 at 10:47 UTC ( #1221972=note: print w/replies, xml ) Need Help??


in reply to Re: Fatal code point 0xFFFFFFFFFFFFFFFF
in thread Fatal code point 0xFFFFFFFFFFFFFFFF

This works ok on many files. I can't really control the input. Is there a way to deal with this issue besides disabling all warnings?
It appears that you are reading a corrupt/illegal sequence of octets from a file which, when fed through Encode::decode(), gets interpreted as a code point greater than the maximum allowed. See "Handling Malformed Data" in the docs for Encode.

Dave.

  • Comment on Re^2: Fatal code point 0xFFFFFFFFFFFFFFFF

Replies are listed 'Best First'.
Re^3: Fatal code point 0xFFFFFFFFFFFFFFFF
by Anonymous Monk on Sep 09, 2018 at 12:43 UTC
    Thanks man but I hate the Encode docs. They can't give one lousy example on the exact syntax of CHECK to suppress unicode's stupid errors. Anyway I tried shoving ,Encode::FB_QUIET in there and it seems to work at turning off the noise. But as usual the docs are not so clear and seem to say that FB_QUIET stops decoding at the error and returns undecoded data? Really not useful. Is just sending Unicode's useless fatals to devnull an effective way to deal with data I just don't care too much about being exactly and perfectly correct according to the commandments of the freakin unicode consortium? Practical extraction of text used to be so easy, now it's calculus :-/

    Thank you for your help

      to suppress unicode's stupid errors.
      They are not stupid errors. The errors (in fact warnings - they only become errors from 5.28 onwards) are telling you that you are trying to feed illegal data into the regex engine. It's illegal because the octets you told decode() to interpret as utf8 aren't in fact valid utf8.

      How you want to handle this corrupt data is of course your choice depending on what is best for your circumstances. You might want to make decode croak if fed a bad file with FB_CROAK, then go back and delete / fixup any bad files. Or depending on the nature of the corruption of the files, you might like to use FB_DEFAULT to just replace the corrupt bits with REPLACEMENT CHARACTER. You would only want to use FB_QUIET if you don't mind decode() stopping at the first bad part of the file and ignoring the rest of its contents.

      Dave.

        Thanks again Dave. I can't believe Perl is gonna die just because of "illegal" unicodes in the regex engine. PERL HAS A LICENSE TO ILL! Why don't they just warn and continue? Or do something mellow like ignore illegal code points, or just change uncool values to the max (and give weenies strict for unicode if they require torture)? Why be Unicode Security Guards? All these negative vibes while parsing something as important as 21st century text can't possibly be right for Perl:
        
        deprecated
        permissible
        fatal
        will not be allowed
        it is too large
        break the perl
        causing it to hang
        problem areas
        breakages
        warning
        upper limit
        out of range
        was deprecated
        a fatal error
        Perl 5.28
        
        
        That's from one error message. I don't know who wrote that but they do not sound like a happy camper! Let's send them some love so they can transform all that dark energy into something wonderful, like this:

        Use of code point 0xFFFFFFFFFFFFFFFF detected; but the permissible max is 0x7FFFFFFFFFFFFFFF. Perl can handle this at X.pm line 1685

        (D dude!) You used a code point that is way too large. Unicode only allows code points up to 0x10FFFF, but of course Perl allows much larger ones. However, the perl interpreter can handle anything. So your error has been automagically corrected! If your code is to run on various platforms, keep in mind that the upper limit depends on the platform. It is much larger on 64-bit word sizes than 32-bit ones.

        The use of out of range code points was acceptable in Perl 5.24, and it will be totally cool, but still pointless, in Perl 5.28.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1221972]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (10)
As of 2019-10-22 11:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Notices?