Re^5: Database vs XML output representation of two-byte UTF-8 character

[Anonymous Monk]

Because it can be used to store any 72-bit values (well, limited to 32- or 64-bit in practice), not just Unicode code points. You've demonstrated this.

(shrug) Yeah, I've never used that feature.

Perl's ability to use a more efficient storage format when possible and a less efficient one when necessary is a great feature, not an awful one. $x = "a"; $x .= "é"; is no more awful than $x = 18446744073709551615; ++$x;. Both cause an internal storage format shift. The lack of ability to tell Perl whether a string is text, UTF-8 or something else is unfortunate because it would allow Perl to catch common errors, but that has nothing to do with the twin storage formats.

Again, I agree and don't agree... The assumption that all strings are in one of the storage formats, unless explicitly specified otherwise, is a source of great confusion. Perl's source code (without "use utf8")? Output of readdir? Contents of @ARGV? I don't see how one can not think about implementation details, storage formats, leaky abstractions and other bad things. To me, 'Perl thinks everything is in Latin-1, unless told otherwise' seems like a more useful, understandable explanation.

Unfortunately, Perl does not have the information it would need to have to know you did something wrong.

For some definitions of 'wrong'. If I actually do have Latin-1 (more realistically, ASCII) than it's not 'wrong', is that what you want to say?

[ikegami]

Again, I agree and don't agree... The assumption that all strings are in one of the storage formats, unless explicitly specified otherwise, is a source of great confusion.

No idea what that means.

Perl's source code (without "use utf8")? Output of readdir? Contents of @ARGV?

Don't know. Don't care. Doesn't matter how they are stored, as those are internal details that aren't relevant.

What does matter is whether they returned decoded text or something else. That has nothing to do with the internal storage format.

To me, 'Perl thinks everything is in Latin-1, unless told otherwise' seems like a more useful, understandable explanation.

It's completely false — nothing in Perl accepts or produces latin-1 — and it has nothing to do with anything discussed so far.

If I actually do have Latin-1 (more realistically, ASCII) than it's not 'wrong', is that what you want to say?

You were complaining that Perl let you concatenate decoded text and UTF-8 bytes. (Well, you called it something different, but this is the underlying issue.) It has no idea one of the the strings you are concatenating contains text and that the other contains UTF-8 bytes, so it can't let you know that you are doing something wrong.

For example,

my $x = chr(0x2660);
my $y = chr(0xC3).chr(0xA9);
$x . $y;

This is all the information Perl currently has. Is that an error? You can't tell. Perl can't tell. Strings coming from a file handle with a decoding layer should be flagged "I'm decoded text!". Those coming from a file handle without a decoding layer should be flagged "I'm bytes!". Concatenating the two should be an error. These flags do not currently exist.

[Anonymous Monk]

No idea what that means.

RTFM then.

"By default, there is a fundamental asymmetry in Perl's Unicode model: implicit upgrading from byte strings to Unicode strings assumes that they were encoded in ISO 8859-1 (Latin-1)"

It's completely false — nothing in Perl accepts or produces latin-1 — and it has nothing to do with anything discussed so far.

LOL. It looks like Latin-1 and quacks like Latin-1, but it's not Latin-1. Yeah, it's just 'byte-packed subset of Unicode'.

"Whenever your encoded, binary string is used together with a text string, Perl will assume that your binary string was encoded with ISO-8859-1, also known as latin-1. If it wasn't latin-1, then your data is unpleasantly converted. For example, if it was UTF-8, the individual bytes of multibyte characters are seen as separate characters, and then again converted to UTF-8."

How about you 'fix' Perl's documentation, and then start arguing... It even talks about 'Unicode' and 'binary' strings (gasp).

my $x = chr(0x2660);
my $y = chr(0xC3).chr(0xA9);
$x . $y;

This is all the information Perl currently has. Is that an error?

Is that an error that perl -wE 'my $x = chr(0x00A9); say $x does one thing, and perl -wE 'my $y = chr(0x2660); say $y' does something else? I dunno. You tell me. intuitively, there should be no difference whatsoever, chr should be consistent, say should be consistent, everything should be... (confused) (not really).

This is all the information Perl currently has. Is that an error? You can't tell. Perl can't tell. Strings coming from a file handle with a decoding layer should be flagged "I'm decoded text!". Those coming from a file handle without a decoding layer should be flagged "I'm bytes!". Concatenating the two should be an error. These flags do not currently exist.

So you're not even disagreeing. You just hate the word 'Latin-1'. I'm done with you. Have a nice day.

[ikegami]

RTFM then.

huh? I asked what you meant.

LOL. It looks like Latin-1 and quacks like Latin-1, but it's not Latin-1. Yeah, it's just 'byte-packed subset of Unicode'.

huh? What are you talking about?

How about you 'fix' Perl's documentation, and then start arguing... It even talks about 'Unicode' and 'binary' strings (gasp).

You can start here. "Perl will assume that your binary string was encoded with ISO-8859-1" is indeed completely wrong. Concatenation does know or care what the string is.

$ perl -MEncode -E'
   $x = chr(0x2660);
   $y = encode($ARGV[0], chr(0xC9));
   say sprintf "%vX.%vX %vX", $x, $y, $x.$y;
' iso-latin-1
2660.C9 2660.C9

$ perl -MEncode -E'
   $x = chr(0x2660);
   $y = encode($ARGV[0], chr(0xC9));
   say sprintf "%vX.%vX %vX", $x, $y, $x.$y;
' UTF-8
2660.C3.89 2660.C3.89

Is that an error that perl -wE 'my $x = chr(0x00A9); say $x does one thing, and perl -wE 'my $y = chr(0x2660); say $y' does something else?

No, Perl "doing something else" (telling you you made an error) when you provide a bad input is not an error.

chr should be consistent,

huh? chr always returns a string consisting of the specified character.

So you're not even disagreeing. You just hate the word 'Latin-1'

huh? What are you talking about?!? No, I hate that you're saying your errors are errors in Perl. I hate that you are spreading misinformation about how Perl works. I hate that you're confusing people with issues that aren't even related to theirs. The OP's problem had nothing to do with internal storage formats.