Re: Malformed UTF-8 character

[Eliya]

Others have explained why Perl complains — in case the string literal is declared with double quotes, at least.  In other words, your source is apparently not encoded in UTF-8, as you're telling Perl with the pragma use utf8.

What I find more surprising is that Perl doesn't complain when - within the scope of use utf8 - the string literal (containing a Latin-1 encoded char like '°') is declared using single quotes.  I'd say the latter is a bug (unless I've overlooked something in the docs... :)

(I can replicate the issue here with 5.12.2.)

[tchrist]

Eliya wrote:

What I find more surprising is that Perl doesn’t complain when — within the scope of use utf8 — the string literal (containing a Latin‑1 encoded char like '°') is declared using single quotes. I’d say the latter is a bug (unless I've overlooked something in the docs... :)

I can confirm it still occurs in 5.14 RC0:

% blead -C0 -le 'print qq(print "\xB0C";)' | blead -Mutf8 -CS -l
Malformed UTF-8 character (unexpected continuation byte 0xb0, with no 
+preceding start byte) at - line 1.
C

% blead -C0 -le 'print qq(print \x27\xB0C\x27;)' | blead -Mutf8 -CS -l
#C

Oops.

[Steve_BZ]

Thanks for this. Do I have the use statements right?

use utf8;
use Encode;
binmode STDOUT, ":utf8";
use open ':encoding(utf8)';

I'm not sure quite what they do or what the difference is. I understand that use utf8 is to save the code page in utf-8, that use Encode is to provide a utility to encode and decode, but I'm not sure what binmode STDOUT, ":utf8";or use open ':encoding(utf8)'; do?

Regards

Steve

[Eliya]

I understand that use utf8 is to save the code page in utf-8

Not really sure what you mean by that... but use utf8 tells Perl that the source code (string literals, etc.) is encoded in UTF-8. So you shouldn't use it if that's not the case for your script.

binmode STDOUT, ":utf8" sets the utf8 PerlIO layer for STDOUT, which tells Perl that you want UTF-8 encoded output for that file handle.

use open ':encoding(utf8)' declares the default layer for I/O streams, i.e. you don't have to explicitly specify the respective layer when you open a file.  See the open pragma for the details.

[Steve_BZ]

Thanks for this.

Regards

Steve