Unpacking variable length bit fields

[puterboy]

I have a bitstream containging 3 length fields and the corresponding 3 fields

The bitstream is layed out as follows:

<length field 1 in bits (1 byte)> <field 1> <length field 2 in bits (1 byte)> <field 2> <length field 3 in bits (1 byte)> <field 3>

I am hoping that there is some clever way to do this with unpack but I don't know how to get the field length while also unpacking without iteratively calling pack.

[BrowserUk]

Assuming your bitfields are all multiples of 8, then using unpack's length/type nomenclature will work.

$bitstream = "\x18\xf0\xff\x0f\x10\xaa\x55\x20\xde\xad\xbe\xef";;

( $f1, $f2, $f3 ) = unpack 'C/b C/b c/B', $bitstream;;

print for $f1, $f2, $f3;;
000011111111111111110000
0101010110101010
11011110101011011011111011101111

You'll need to use the correct variations of 'c' or 'C' (probably the latter) and b/B (which will depend upon the endianess of the originating hardware. I've used a mixture here to demonstrate the affect of some of the combinations.

Note also that the output is ascii-ised binary. That is, each bit of the input is translated to a byte containing ascii '0' or ascii '1'.

Other variations are possible. For example, you might want to retain the bits as binary encoded fields and then manipulate them with vec. But you'd need to explain more about what they are.

[puterboy]

Oops, I spoke to soon. The field length specifies the number of bits (multiple of 8) but I want to extract the fields themselves as bytes not as bit strings of 1/0's.

My problems is that using something like: unpack('C/C'...) would unpack C bytes for each filed rather than C bits for each field which is what I want. So I really would like to divide by 8 but I'm not sure how to do that. I imagine I could unpack as a bit string of 1/0's then split, then repack but that seems klugey.

It's a screwy format, but I don't have control over it....

[BrowserUk]

The the easiest way is to convert the ascii-ised binary back to real numbers:

$bitstream = "\x18\xf0\xff\x0f\x10\xaa\x55\x20\xde\xad\xbe\xef";;

( $f1, $f2, $f3 ) = map unpack( 'L', pack( 'b32', $_ )), unpack 'C/b C
+/b C/b', $bitstream;;

print for $f1, $f2, $f3;;
1048560
21930
4022250974

This assumes maximum 32-bit values. You could use Q if you are on a 64-bit platform and Perl and need that.

By converting them all to the maximum sized integer the platform can handle you save having to make decisions and they'd end up that way anyway.

You might need to adjust the L (or the 'b32' to 'B32' or 'B64') for the endianness of the originating platform.

[puterboy]

This appears to be EXACTLY what I needed. Thanks!

[ikegami]

If your lengths aren't necessarily multiples of 8,

my @fields;
my $bits = unpack('B*', $input);
while ($bits) {
   die if length($bits) < 8;
   my $num_bits = unpack('C', pack('B*', substr($bits, 0, 8, '')));
   my $field = substr($bits, 0, $num_bits, '');
   push @fields, $field;
}

You can probably do this more efficiently using bit twiddling, but this is simpler.

If you want the result as a number rather than a string of "1"s and "0"s, you can use the following:

use Config qw( );
use constant UV_BITS => $Config::Config{uvsize} * 8;

die if $num_bits > UV_BITS;
push @fields, pack('B*', substr(("0" x UV_BITS).$field, -UV_BITS));

[Limbic~Region]

puterboy,
Assuming this data is in a string, I would do something like:

my $skip = 0;
while ($skip < length($str)) {
    my ($f1_len, $f2_len, $f3_len) = unpack("x${skip}aaa", $str);
    $skip += 3;
    my $field1 = unpack("x${skip}a$f1_len", $str);
    $skip += $f1_len;
    # etc
}

This is like using seek which is exactly what you were trying to avoid (iteratively calling pack) but if wrapped in the proper abstraction can be very clean.

Update: After reading BrowserUk's response and re-reading your post, I realize I incorrectly assumed you had multiple records in the string - each one with variable length fields. Please ignore me.