Re^2: How to reverse a (Unicode) string

[moritz]

Note that iso-8859-1 is a unicode encoding -- it just doesn't support all of the characters.

I don't know what you mean by "unicode encoding" (are there encodings that map to non-unicode chars?), but in the perl context it's worth mentioning that iso-8859-1 strings don't follow unicode-semantics by default, the need to be encoded like any other string:

# this file is stored as latin1 
print "ä" =~ m/\w/ ? "Unicode\n" : "Bytes\n";
__END__
Bytes

Perl's unicode support is great, but the programmer MUST learn the difference between unicode and utf-8, and the difference between text data and binary data.

Yes, and they have to learn that for any kind of tool that supports Unicode and different encodings.

And I really like the Perl 6 spec which allows string operations on byte, codepoint and grapheme level ;-)

[Juerd]

I don't know what you mean by "unicode encoding" (are there encodings that map to non-unicode chars?), but in the perl context it's worth mentioning that iso-8859-1 strings don't follow unicode-semantics by default, the need to be encoded like any other string

It is a unicode encoding, in that after you've decoded the character number, the number maps 1-on-1 to the Unicode space. Don't forget that UTF-8 is just a way of encoding a sequence *numbers*.

That non-SvUTF8-flagged strings get ASCII semantics in some places, is indeed by design, but that wasn't sufficiently thought through IMO. Note that these strings may get unicode semantics in some circumstances, and ascii semantics in others. The ascii semantics are for charclass and upper-/lower case stuff.

I consider this a bug in Perl. See also Unicode::Semantics, and expect the bug to be fixed in 5.12.

And I really like the Perl 6 spec which allows string operations on byte, codepoint and grapheme level ;-)

Just realise that Unicode strings don't have a byte level :)

[moritz]

I hesitated before writing about the byte level, but S02 explictly mentions it:

You can also ask for the total string length of an array's elements, in bytes, codepoints or graphemes, using these methods .bytes, .codes or .graphs respectively on the array. The same methods apply to strings as well.

And of course you have a byte level if you specify an encoding, or if there is a default one. Just like you can have a language dependent notion of a grapheme if you pick a language.

But I think the spec should be a bit clearer regarding how the character encoding is chosen.

[Juerd]

The way I expect it to work, is that Unicode strings don't get encodings, but byte buffers and I/O handles do. If you give a Buf an encoding, you can treat it as a Str, as long as the encoding is consistent. Using a real Str may perform much better, because Perl 6 can encode it internally however it wishes, and can make many more assumptions (for optimization purposes).

Giving a Unicode string an "encoding" property is the wrong way around, but it can take quite some time and experience before one realises that.

[ikegami]

[iso-8859-1] is a unicode encoding, in that after you've decoded the character number, the number maps 1-on-1 to the Unicode space.

By that logic, UTF-8 is not a "unicode encoding". For example, C2 in Unicode does not map to C2 in UTF-8. Your choice of name for this trait is very poor.

[JavaFan]

You seem to be confusing "1-to-1" mapping, and "identity function". While the identity function is a trivial "1-to-1" mapping, it's not true every "1-to-1" mapping is the identity function.

However, even side-stepping that, Juerd doesn't mean byte values map 1-to-1. The mapping is after decoding. For instance, the UTF-8 byte sequence 0x82 0xC3 decodes to C2. Which indeed does map to the C2 Unicode code point.