http://www.perlmonks.org?node_id=661537


in reply to Re^2: How to reverse a (Unicode) string
in thread How to reverse a (Unicode) string

I don't know what you mean by "unicode encoding" (are there encodings that map to non-unicode chars?), but in the perl context it's worth mentioning that iso-8859-1 strings don't follow unicode-semantics by default, the need to be encoded like any other string

It is a unicode encoding, in that after you've decoded the character number, the number maps 1-on-1 to the Unicode space. Don't forget that UTF-8 is just a way of encoding a sequence *numbers*.

That non-SvUTF8-flagged strings get ASCII semantics in some places, is indeed by design, but that wasn't sufficiently thought through IMO. Note that these strings may get unicode semantics in some circumstances, and ascii semantics in others. The ascii semantics are for charclass and upper-/lower case stuff.

I consider this a bug in Perl. See also Unicode::Semantics, and expect the bug to be fixed in 5.12.

And I really like the Perl 6 spec which allows string operations on byte, codepoint and grapheme level ;-)

Just realise that Unicode strings don't have a byte level :)

  • Comment on Re^3: How to reverse a (Unicode) string

Replies are listed 'Best First'.
Re^4: How to reverse a (Unicode) string
by moritz (Cardinal) on Jan 10, 2008 at 08:33 UTC
    I hesitated before writing about the byte level, but S02 explictly mentions it:

    You can also ask for the total string length of an array's elements, in bytes, codepoints or graphemes, using these methods .bytes, .codes or .graphs respectively on the array. The same methods apply to strings as well.

    And of course you have a byte level if you specify an encoding, or if there is a default one. Just like you can have a language dependent notion of a grapheme if you pick a language.

    But I think the spec should be a bit clearer regarding how the character encoding is chosen.

[reply]
[d/l]
[select]

      The way I expect it to work, is that Unicode strings don't get encodings, but byte buffers and I/O handles do. If you give a Buf an encoding, you can treat it as a Str, as long as the encoding is consistent. Using a real Str may perform much better, because Perl 6 can encode it internally however it wishes, and can make many more assumptions (for optimization purposes).

      Giving a Unicode string an "encoding" property is the wrong way around, but it can take quite some time and experience before one realises that.

[reply]
Re^4: How to reverse a (Unicode) string
by ikegami (Patriarch) on Jan 09, 2011 at 23:52 UTC

    [iso-8859-1] is a unicode encoding, in that after you've decoded the character number, the number maps 1-on-1 to the Unicode space.

    By that logic, UTF-8 is not a "unicode encoding". For example, C2 in Unicode does not map to C2 in UTF-8. Your choice of name for this trait is very poor.

[reply]
      You seem to be confusing "1-to-1" mapping, and "identity function". While the identity function is a trivial "1-to-1" mapping, it's not true every "1-to-1" mapping is the identity function.

      However, even side-stepping that, Juerd doesn't mean byte values map 1-to-1. The mapping is after decoding. For instance, the UTF-8 byte sequence 0x82 0xC3 decodes to C2. Which indeed does map to the C2 Unicode code point.

[reply]

        In that case, we're back to the original question. Are there any encodings aren't "Unicode encodings"?

        (Strictly speaking, the mapping isn't 1-to-1. U+2660 can't be encoded in iso-8859-1. You could also say that both U+00E9 and U+0065 U+0301 encode to E9 in iso-8859-1, although Encode's encode doesn't handle that.)

[reply]
[d/l]
        No, actually, I'm not confused. When the term was introduced, it was given as the reason iso-8859-1 works without being decoded, so he indeed meant an identity mapping.
[reply]

In Section Tutorials