Re^4: Database vs XML output representation of two-byte UTF-8 characterby ikegami (Pope)
|on Sep 07, 2014 at 16:49 UTC||Need Help??|
Very few need to know about Perl internals that are irrelevant to the problem at hand.
Because it can be used to store any 72-bit values (well, limited to 32- or 64-bit in practice), not just Unicode code points. You've just demonstrated this.
Perl's ability to use a more efficient storage format when possible and a less efficient one when necessary is a great feature, not an awful one. $x = "a"; $x .= "é"; is no more awful than $x = 18446744073709551615; ++$x;. Both cause an internal storage format shift.
The lack of ability to tell Perl whether a string is text, UTF-8 or something else is unfortunate because it would allow Perl to catch common errors, but that has nothing to do with the twin storage formats.
That would be helped by the aforementioned type system, but not by misunformation.
Unfortunately, Perl does not have the information it would need to have to know you did something wrong.
It does warn you when it knows a problem occurred (as you mentioned), but it can't warn when it doesn't know.
Those that work?
There's definitely room for improvement, I'm not disputing that.