Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

converting artibrary binary data to numeric values

by markle (Initiate)
on Sep 30, 2012 at 18:00 UTC ( #996521=perlquestion: print w/replies, xml ) Need Help??
markle has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

Give me all the abstract database tasks you like, but I feel like I should wear a dunce cap when it comes to binary data.

I'm getting an HDLC-encoded stream from a device. I can decode it okay. Each packet has a format which contains header information followed by binary-packed payloads of various types and sub-types. These are defined from the device's C/C++ code.

I unpack the bytes using the 'a' template for arbitrary binary data. I can view the data in hex and it corresponds to the byte order of the C struct. However, I don't know what to do to convert some of those binary values to numbers like uint32 or int32 values.

For example, one of the uint32 fields in the C struct is a binary flag. The hex value is 0000 0001. Perl says the value is a string, "\0\0\0\x01". How do I convert this to a numeric value without resorting to unpacking each byte and bit-shifting?

Dunce cap. :-) -Mark

Replies are listed 'Best First'.
Re: converting artibrary binary data to numeric values
by Corion (Pope) on Sep 30, 2012 at 18:08 UTC

    That's exactly what unpack (and pack for the documentation) are for.

    Depending on whether your sending device sends in big-endian or little-endian order, the N or V template elements look like you want them.

    Also, if you have the C++ source code of the remote end, consider Struct::Binary::C, which was written for importing C structs sensibly into Perl.

      Like with unpacking from numbers it seems Perl first converts the raw binary data into a string representation of a gigantic decimal number, then it tries to unpack from that, giving bizarre results that do not correspond to the original byte order.

      Also, it seems the "b" and "B" flags to unpack expect the value to be a string of 1's and 0's, not a number.

      Please help me answer my original question. How do I convert a value which perl thinks is the scalar "\xd\xe\xa\xd\xb\xe\xe\xf" into a scalar with the numeric value 0xDEADBEEF ?

        Are you really, really sure that you get eight bytes from your file, the first having the value 0x0D and the second having the value 0x0E (and so on)?

        If so, you might need to do some bit twiddling to combine all the nibbles:

        use Data::Dumper; my $str = "\x0D\x0E\x0A\x0D\x0B\x0E\x0E\x0F"; my @bytes = unpack 'C8', $str; print Dumper \@bytes; # Now add up the byte-pairs for (0..3) { $bytes[ $_ ] = $bytes[ $_*2 ] * 16 + $bytes[ $_*2 +1 ]; }; splice @bytes, 4; # we only need the first four entries print Dumper \@bytes; my $num = unpack 'N', pack 'C4', @bytes; print sprintf "%08x (%i)", $num, $num;

        But if you get a string of four bytes, one of the two pack templates "N" or "V" will help you. See the following:

        my $str = "\xDE\xAD\xBE\xEF"; my $num = unpack 'N', $str; print sprintf "%08x (%i)", $num, $num;

        But I already told you this and you didn't show any code.

Re: converting artibrary binary data to numeric values
by johngg (Canon) on Sep 30, 2012 at 21:57 UTC

    This might help you visualise what is happening when packing and unpacking big- and little-endian values, as mentioned in Corion's reply.

    $ perl -E ' > say q{Big-endian}; > $BEflag = pack q{N}, 1; > say unpack q{B*}, $BEflag; > say sprintf q{0x%02x}, ord $_ for split m{}, $BEflag; > say unpack q{N}, $BEflag; > say q{Little-endian}; > $LEflag = pack q{V}, 1; > say unpack q{B*}, $LEflag; > say sprintf q{0x%02x}, ord $_ for split m{}, $LEflag; > say unpack q{V}, $LEflag;' Big-endian 00000000000000000000000000000001 0x00 0x00 0x00 0x01 1 Little-endian 00000001000000000000000000000000 0x01 0x00 0x00 0x00 1 $

    I hope this is helpful.



Re: converting artibrary binary data to numeric values
by GrandFather (Sage) on Oct 01, 2012 at 23:29 UTC

    Show us some code that demonstrates the problem. Corion has shown various sample code that probably answers your question, but I suspect you don't understand it so you are guessing at answers and are not only getting it wrong, but you are probably providing misleading information to us too.

    So, show us some sample code using one of the two strings "\x0D\x0E\x0A\x0D\x0B\x0E\x0E\x0F" or "\xDE\xAD\xBE\xEF" (whichever is appropriate) and tell us what you get and what you expect. Note that the second string is most likely to be correct. Both strings are big endien.

    For your original question the code is simply my $flags = unpack('N', "\0\0\0\x01");

    True laziness is hard work

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://996521]
Approved by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (5)
As of 2018-10-20 10:54 GMT
Find Nodes?
    Voting Booth?
    When I need money for a bigger acquisition, I usually ...

    Results (117 votes). Check out past polls.

    • (Sep 10, 2018 at 22:53 UTC) Welcome new users!