http://www.perlmonks.org?node_id=948422


in reply to Jargon relating to Perl strings

Under Basics
The sequence of string elements in a string. This is not affected by the string's UTF8 flag.
This is not what you want to say (it immediately confused me because my first thought was, "This is wrong because if you have a string with non-ASCII characters in it and change its UTF8 flag, that will change the sequence of elements.")

What I think you meant to say

The sequence of string elements in a string, irrespective of any particular choice of memory representation being used for that string
and only bring up the UTF8 flag later.

Also, IMHO there needs to be distinct terminology for

  1. a grouping of (usually but not always) 8 consecutive bits of physical storage
  2. the abstract array element in the case where all elements are expected to be in the range 0-255, regardless of the actual storage format
the problem being (as noted by others) that most people associate "byte" with (1). Using it for (2) is unlikely to reduce the confusion out there and not having distinct terminology makes it difficult to talk about storage formats.

One could possibly commandeer "octet" for (2) but realize that "octet" originated in the RFC world where a word was needed to refer to physical storage in the specific case where bytes explicitly are known to be 8 bits. On the other hand, most of the stuff in the RFC world is indeed trying to abstract away from specific hardware, so one could justify its usage in a more abstract sense that way. And "octet" does, at least, immediately imply 0-255, unlike "byte"

There's also the small matter that it really doesn't make a whole lot of sense to use UTF-8-flag-on format to store something that is composed of octets, even if it is indeed possible to do. Which is why people conflate octet strings with the UTF-8-flag-off format and its 1-1 correspondence between octets and bytes
(...and thus why it is indeed important to point out that UTF-8-flag-on octet strings are possible, albeit silly...)

Update: yeah I edited this... sorry.

Replies are listed 'Best First'.
Re^2: Jargon relating to Perl strings
by ikegami (Patriarch) on Jan 18, 2012 at 02:22 UTC

    This is not what you want to say

    Indeed. I discovered this reading JavaFan's post.

    What I think you meant to say

    Perfect. I shall update.

    a grouping of (usually but not always) 8 consecutive bits of physical storage

    UTF8=0 storage format.

    the problem being (as noted by others) that most people associate "byte" with (1)

    If so, then reading 5 bytes produces a variable number of bytes*. That doesn't jive.

    If I read 5 bytes from a file, what I get if 5 bytes as far as I'm concerned. I'm open to a better word that "byte" for this, but I haven't come across one.

    One could possibly commandeer "octet"

    An octet is simply an 8-bit byte. But since that's what byte means in all relevant circumstances anyway*, "octet" is no better than "byte".

    * — Perl doesn't currently support systems with byte sizes other than 8.

    There's also the small matter that it really doesn't make a whole lot of sense to use UTF-8-flag-on format to store something that is composed of octets, even if it is indeed possible to do.

    No, but it can happen. Say you have: 

    # É as C9 in source.
print $bin_fh "AX100ÉX";

    [download]

    And say one day you convert your source files to UTF-8. 

    # É as C3 E9 in source.
use utf8;
print $bin_fh "AX100ÉX";

    [download]

    The code is still fine, yet the string in the latter has UTF8=1.

[reply]
[d/l]
[select]
      Perfect. I shall update.
      except you updated the wrong thing. It's the sentence "This is not affected by the string's UTF8 flag," under Basics->"String Value" that's tripping me (and apparently also javafan) up and that needs to either go away or be changed
      a grouping of (usually but not always) 8 consecutive bits of physical storage
      UTF8=0 storage format.
      No, any storage format. In order to talk about storage formats at all you need a word for the raw underlying bytes whatever they are and however they're to be interpreted, and redefining "byte" to mean something else makes this really difficult.

      You need a different word, and you're probably right that "octet" isn't a great choice either, so I had another thought: How about one of the following to refer to string elements that are constrained to lie in the 0-255 range?

      • "octetchar"
      • "octet-character"
      • "bytecharacter"
      • "byte-character"
      • "bytechar"
      as opposed to "general character" or "Unicode character" when the full Unicode (or UV) range is possible. This way you're emphasizing that they're still characters in the sense that everybody agrees on (i.e., they're elements of a string and we're abstracting away from how they're represented). If you then say that a single octetchar can actually be multiple bytes in the UTF8=1 storage format, the meaning is clear.
[reply]

        except you updated the wrong thing.

        I had *missed* one of the three spots where I used the problem phrase. Fixed the one I have missed.

        In order to talk about storage formats at all you need a word for the raw underlying bytes whatever they are and however they're to be interpreted,

        It's not really something that comes up, but "contents of the PV".

[reply]

In Section Meditations