Re^3: Character in 'b' format wrapped in unpack

[graff]

With all due respect for your justifiable anger, I'm sorry to disagree; "chr()" is - and rightly should be - intended to serve the (dominant) linguistic sense of "character" (what the perl docs call "character semantics"), rather than the strictly-typed, C-centric sense of "char" (what the perl docs call "octets" or "byte semantics").

In other words, when you want to do low-level, C-like bit twiddling, just use pack and unpack - that's what those are for - and give up on pretending that higher-level, linguistically oriented functions (chr and ord) can do the same thing.

I agree how sad it is that every user must pay the performance cost of unicode support, whether or not they actually need or use it. But then, it's also sad that every script must pay the overhead for untyped variables, no matter how much of that flexibility is actually needed or used.

UPDATE: Having said that, I realize I'm probably still deficient in my understanding of your particular example. You said you "wanted the shift to discard the high bit, as it does with integers", and if I'm not mistaken (am I?), that's actually what happens, with or without the "use bytes" pragma (i.e. with or without the warning). Here's a simpler example - am I missing something?

#use bytes;

$x = "\xAA";
print unpack 'B*', $x;
print " --> ";
$x = chr( ord( $x ) << 1 );
print unpack 'B*', $x;

print "\n---\n";

$x = pack( "C*", 0xAA );
print unpack 'B*', $x;
print " --> ";
$x = pack( "C", unpack( "C", $x ) << 1 );
print unpack 'B*', $x;

print "\n---\n";

When I run that, I get:

Character in 'B' format wrapped in unpack at /tmp/j1.pl line 7.
10101010 --> 01010100
---
Character in 'C' format wrapped in pack at /tmp/j1.pl line 14.
10101010 --> 01010100
---

(Note the warning from using the "C" format on pack.) Looks to me like the high bit got shifted off in both cases - no difference. When I uncomment the "use bytes", the only difference I see is that the "B" format warning goes away (but the "C" format warning still shows up.) Is there a problem I'm not seeing?

In case it matters, I'm using perl 5.18 on macosx 10.10.2 ("yoesemite").

[BrowserUk]

I'm sorry to disagree

I guess we'll have to agree to differ; but the fact that Perl allows me to replace an (8-bit) character, in the middle of a string of 8-bit characters, with some (random*) wide character is just broken.

"chr()" is - and rightly should be - intended to serve the (dominant) linguistic sense of "character" (what the perl docs call "character semantics")

To what possible end?

When you do my $thing = chr( 12345 ); what does that "character" represent?

Is a Chinese character? Or Sanskrit? Or Cyrillic?

Is it utf-8; utf16; utf32?

Is it big-endian or little-endian?

What if I append another character to it: $thing .= chr( $i );. What does string contain now? Can Perl ever decide what encoding $thing contains?

And the answer to all of those questions is: it is impossible to ever know. Thus, chr's ability to construct wide characters is entirely useless.

So, you break with clearly defined semantics for undefined and undefinable semantics, for what purpose?

[choroba]

When you do my $thing = chr( 12345 ); what does that "character" represent?

Is a Chinese character? Or Sanskrit? Or Cyrillic?

Is it utf-8; utf16; utf32?

Is it big-endian or little-endian?

It's Unicode. It's HANGZHOU NUMERAL TWENTY, in fact. UTF-8, UTF-16 both represent unicode codepoints, but encode them differently.

When you concatenate a different string to it, the result might depend on the version of Perl. See unicode_strings.

[BrowserUk]

It's Unicode.

Great! Then this must be unicode also:

perl -MDevel::Peek -E"$x = chr(129).chr(130).chr(42).chr(131).chr(132)
+; Dump($x); substr( $x, 2, 1 ) = chr(~0); Dump($x); print $x" | od -t
+x1
SV = PV(0xbadc0) at 0x2c5aa8
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0xb67a8 "\201\202*\203\204"\0
  CUR = 5
  LEN = 8
SV = PVMG(0x2b1078) at 0x2c5aa8
  REFCNT = 2
  FLAGS = (SMG,POK,pPOK,UTF8)
  IV = 0
  NV = 0
  PV = 0x2b3008 "\302\201\302\202\377\200\217\277\277\277\277\277\277\
+277\277\277\277\302\203\302\204"\0 [UTF8 "\x{81}\x{82}\x{ffffffffffff
+ffff}\x{83}\x{84}"]
  CUR = 21
  LEN = 24
  MAGIC = 0x3177f8
    MG_VIRTUAL = &PL_vtbl_utf8
    MG_TYPE = PERL_MAGIC_utf8(w)
    MG_LEN = -1
Wide character in print at -e line 1.
0000000 c2 81 c2 82 ff 80 8f bf bf bf bf bf bf bf bf bf
0000020 bf c2 83 c2 84
0000025

Broken!

[ikegami]

"Character" just means "string element". In C, they are usually 8 (but 9 bits and other widths are possible). In Perl, they are far bigger. In both languages, they are numbers devoid of intrinsic meaning. They can be all of the things you specified, or something completely different.

[BrowserUk]

but 9 bits and other widths are possible

Oh yeah! Then why didn't they just create a 20.0625-bit character type to hold the 1,114,112 possible codepoints?