How scalars work (about numbers, text strings, and binary strings)


This is a simplified high level view of relations between Perl scalar types.

                             ┌───────────────┐
                      ┌───── │ REFERENCE     │ ─────┐
                      │      │ (ROK flag on) │      │
                      │      └───────────────┘      │
      numeric context │                             │ string context
                      │                             │
                      ▼                             ▼
┌─────────────────────────┐                     ┌──────────────────────────────┐
│ NUMBER                  │    string context   │ TEXT STRING                  │
│ encoded internally      │ ──────────────────▶ │ (POK flag on)                │
│ as any of:              │                     │ encoded internally           │
│ * integer (IOK flag on) │   numeric context   │ as one of:                   │
│ * double (NOK flag on)  │ ◀────────────────── │ * iso-8859-1 (UTF8 flag off) │
└─────────────────────────┘                     │ * utf8 (UTF8 flag on)        │
                  │     ▲                       └──────────────────────────────┘
                  │     │                         ▲     │           ▲
                  │     │                         │     │           │
             pack │     │ unpack           decode │     │ encode    │
                  │     │                         │     │           │
                  ▼     │                         │     ▼           │ :encoding
                ┌─────────────────────────────────────────┐         │ PerlIO
                │ BINARY STRING                           │         │ layer
                │ (POK flag on)                           │ ◀───────┘
                │ (UTF8 flag off)                         │
                └─────────────────────────────────────────┘
                        ▲  ▲  ▲
                        │  │  │
                        │  │  │
                        ▼  ▼  ▼
                  ┌─────────────────────────────┐
                  │ OUTSIDE PERL                │
                  │ files, sockets, filenames,  │
                  │ environment, system calls   │
                  └─────────────────────────────┘

(A Perl programmer does not have to know about the internal flags ROK, IOK,
NOK, POK, and UTF8, but if you're interested read perlguts.)


Keep text and binary strings/semantics separated! (Good style anyway!)

If you don't keep them separate, and use a binary string as a text string, it
is assumed to be iso-8859-1 encoded.

If you don't keep them separate, and use a text string as a binary string, one
of the following things happens, with or without warnings:

  1. the internal iso-8859-1 buffer is used (always the case if the internal
     buffer is not utf8 encoded)
  2. the internal utf8 buffer is used
  3. the iso-8859-1 encoded version is used
     3a. characters above U+00FF are utf8 encoded, while the rest is iso
     3b. characters above U+00FF are modulo'ed 256
     3c. characters above U+00FF are dropped
     3d. characters above U+00FF cause an exception to be thrown

If you do keep them separate, and always explicitly convert between the two
types by explicitly decoding and encoding or using the :encoding layer on a
filehandle, you stay in control of what happens and your program will behave
more predictably.

Update: thin lines used, see discussion below.

Comment on How scalars work (about numbers, text strings, and binary strings)

Replies are listed 'Best First'.
Re: How scalars work (about numbers, text strings, and binary strings) by ikegami (Pope) on Oct 17, 2007 at 15:54 UTC
One way to keep binary strings seperate from text strings when you have to deal with both is to use hungarian notation. `my $bin_msg = ...; my $txt_msg = decode($encoding, $bin_msg);` [download] See Making Wrong Code Look Wrong for more on the subject. (I usually use "bytes" and "chars". I used "bin" and "txt" here to be in line with your diagram.)	[reply] [d/l]
Re^2: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Oct 17, 2007 at 18:56 UTC
That's a great idea, and I have done it in a few examples. I'd have to fight my old habits in order to do such a thing in actual code, though. Maybe it would be easier to do with single-letter prefixes. "u" for Unicode, perhaps. Feels better than "t" or "c" (I read "c" as "count", and would expect a number). Juerd # { site => 'juerd.nl', do_not_use => 'spamtrap', perl6_server => 'feather' }	[reply]
Re: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Oct 17, 2007 at 15:41 UTC
Here's a screenshot for people with broken browsers that don't do the Unicode line drawing characters correctly: http://juerd.nl/files/perlvalues.png	[reply]
Re^2: How scalars work (about numbers, text strings, and binary strings) by halley (Prior) on Oct 19, 2007 at 14:23 UTC
In my Firefox (not an obscure browser), this diagram formatted incorrectly, causing the whole column to expand twice as wide as intended to the right. If you include text or diagrams that might screw up formatting, can you please put it in `<readmore>` tags? -- `[ e d @ h a l l e y . c c ]`	[reply] [d/l]
Re^3: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Oct 19, 2007 at 19:41 UTC
Sorry, your browser is broken beyond what I personally care to work around. The post is a mere <pre> tag with text in it, no further formatting applied. If even this breaks, anything might, and we could wrap the entire internet in <readmore>... Please check and repair your setup. The 80 characters width is a conscious conservative choice, and the diagram renders correctly on my computer in Firefox, Opera, Konqueror, Internet Explorer, Safari, w3m, Links2, Lynx, Dillo, ... That's all the major browsers and a few obscure ones. I had reports of some monospace fonts being not-so-monospaced for unicode linedrawing characters (while especially there, it really matters), so I uploaded the screenshot at http://juerd.nl/files/perlvalues.png. I try my best, but I have to draw a line with regards to how far I want to go, and adding <readmore> tags to hide the most important part of this work is well beyond that line. Juerd # { site => 'juerd.nl', do_not_use => 'spamtrap', perl6_server => 'feather' }	[reply]
Re^4: How scalars work (about numbers, text strings, and binary strings) by ikegami (Pope) on Oct 19, 2007 at 20:05 UTC
Re^5: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Oct 19, 2007 at 21:29 UTC
Re^4: How scalars work (about numbers, text strings, and binary strings) by jdporter (Canon) on Oct 19, 2007 at 20:39 UTC
Re^5: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Feb 10, 2009 at 21:16 UTC
Re: How scalars work (about numbers, text strings, and binary strings) by Joost (Canon) on Oct 19, 2007 at 21:35 UTC
Nice work. Now, back to the discussion of the technicalities of the diagram :-) What program did you use to draw it? I've just discovered emacs artist mode a few weeks ago and it's quite nice, but it appears to only do basic ASCII drawings. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re^2: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Oct 19, 2007 at 21:38 UTC
What program did you use to draw it? Vim, with the help of The Unicode Sliderule, and copy/pasting. I used the same thing, plus regexes, to change the thick lines to thin lines. I first modelled it on paper, then in Inkscape. It started out as a rather complex diagram, but the end result is, as you can see, quite simple and comprehensible. Update: your post inspired me to look for tools that can help, and I found this nifty Vim script: http://www.vim.org/scripts/script.php?script_id=173 Juerd # { site => 'juerd.nl', do_not_use => 'spamtrap', perl6_server => 'feather' }	[reply]
Re: How scalars work (about numbers, text strings, and binary strings) by Juerd (Abbot) on Oct 22, 2007 at 12:21 UTC
Note that in reality, references go straight to binary strings when stringified. This is a bug and likely to be fixed some version after 5.10.0.	[reply]