http://www.perlmonks.org?node_id=692238

Spooky has asked for the wisdom of the Perl Monks concerning the following question:

Hello, This is my first PERL question and probably won't be my last! I have a binary file and when I access it with emacs using the "hexl-mode" option I'm presented with the following: 0000 0034 0000 31b1 0191 403b 8811 bb13 66e4 formatted as 4bytes 4bytes 1byte 1byte 8bytes which translates into the following numbers: 52 12721 1 145 27.53152055 I need a way to convert the hex values into either integer or floating point using PERL - any suggestions?

Replies are listed 'Best First'.
Re: HEX to floating point
by almut (Canon) on Jun 16, 2008 at 13:14 UTC

    As to the floating point number: 

    my $hex = "403b8811bb1366e4";
print unpack "d", reverse pack "H*", $hex;  # 27.53152055 (on x86 arch
+)

    [download]

    but, as moritz said, binary representations of numbers are machine dependent... (except single-byte ints), so depending on the architecture that you're running this on, you may get wrong results (note that "d" is "A double-precision float in the native format").   (Update: in practice, however, most architectures these days use IEEE 754 format, so you essentially only have to figure out whether little- or big-endian representation is being used — handled by the reverse in the above snippet.)

    Also, in case you're reading the data from the file in binary (not as a hex string), you of course don't need the pack "H*" part shown above.

[reply]
[d/l]
[select]
Re: HEX to floating point
by BrowserUk (Patriarch) on Jun 16, 2008 at 13:47 UTC

    You can change the endianess of a whole string of binary data using scalar reverse, so long as you remember to reverse the template and results: 

    $raw = pack 'H*',  '00000034000031b10191403b8811bb1366e4';;

print for reverse unpack 'dCCVV', reverse $raw;;
52
12721
1
145
27.53152055

    [download]

    If you're working with large volumes of binary structures, doing it this way can save quite a lot of time.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      This is what I'd like to be able to do but when I entered your code into a file with the PERL load module I didn't get anything like your output - I did get a huge floating point number! ..any suggestions - I am new at PERL as if you didn't already guess... Here's my file: #!/fs/COTS/gnu/bin/AIX/perl $raw = pack 'H*", '00000034000031b110191403b8811bb1366e4'; print for reverse unpack 'dCCVV', reverse $raw ; ..when I run this (simply type "test.pl") I get : 1331232565922520241.709119561536
[reply]

        First, you have a typo in your hex string: '00000034000031b110191403b8811bb1366e4' (the highlighted 1).  Second, if you run this on AIX (as I figure from the presence of "AIX" in the shebang path), you most likely don't want the reverse stuff anyway, as AIX typically runs on big-endian hardware, which already matches the binary format of your numbers...

        In other words, on AIX, this should work (actually it does — tried it): 

        my $raw = pack "H*", '00000034000031b10191403b8811bb1366e4';
print "$_\n" for unpack 'NNCCd', $raw;

        [download]
[reply]
[d/l]
[select]
        I entered your code into a file with the PERL load module

        Hm. I'm afraid I have no idea what that means? What is the "Perl load module"?

        Perhaps you could cut & paste the terminal session to show what you did?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: HEX to floating point
by tachyon-II (Chaplain) on Jun 16, 2008 at 14:13 UTC

    You have an endianness issue. You are big endian. On my little endian Wintel machine: 

    # $h = unpack "H16", pack "d", 27.53152055;
27.53152055  => e566 13bb 1188 3b40  

Now you have => 403b 8811 bb13 66e4

# $h = reverse unpack "h16", pack "d", 27.53152055;
# 403b8811bb1366e5

    [download]

    I'll leave you to sort out the why's and wherefores and just give you some code to play with. You may well need to fiddle with it on your machine as "d" is native format double and your native format is big endian, unlike mine. By fiddle I mean try other templates 'h' and 'H' are similar but subtly different. 'N' and 'V' are big endian versus little endian 32 bit ints.

    I have unpacked the ints into an "N" which is a long in "Network" (big endian) order which seems to be what you have given the results you desire. Also unless you have a 64 bit Perl and can therefore unpack using a quad (64 bit int) "q/Q" pack template you will need to use something like Math::BigInt if you need 64 bit ints. At the moment the hex2int sub just takes the least significant 32 bits and silently discards any higher order bits. 

    sub fix_endian {
    my $h = shift;
    my $h = reverse $h;
    $h =~ s/(.)(.)/$2$1/g;  # only required if using 'H' not 'h' templ
+ate
    return $h;
}
sub hex2int   { unpack("N", pack("H8", substr('0'x8 .$_[0], -8))) }
sub hex2float { unpack("f", pack("H8", substr('0'x8 .$_[0], -8))) }
sub hex2double{ unpack("d", pack("H16", substr('0'x16 .$_[0], -16))) }

while(<DATA>){
    s/\s+//g;
    next unless $_;
    $, = "\t";
    print $_, hex2int($_), hex2float($_), hex2double($_), hex2double(f
+ix_endian($_)), "\n";
}

__DATA__
0000 0034 
0000 31b1
01
91
403b 8811 bb13 66e4

    [download]
[reply]
[d/l]
[select]

      I realize those subs contain a trivial amount of code but a packtard like me sure wouldn't mind seeing those gathered together in a module with a nice side of Pod. (Think of the fame, wealth, and women that could be yours!)

[reply]

        Sadly they are not portable due to the endianness of the source data and then the endianness of the destination reader system. You can find 3 diffferent ways to unpack hex strings in faq4 from memory (using hex, pack/unpack, and Bit::Vector])

        As you may have noted BrowserUk presented a method that converts the data given from the OPs system (big endian) to his system (little endian wintel). When the OP tried to run it he found it broken.

        Pack/Unpack Tutorial (aka How the System Stores Data) by pfaut is an excellent resource for understanding pack/unpack.

[reply]
Re: HEX to floating point
by moritz (Cardinal) on Jun 16, 2008 at 12:29 UTC
    perlpacktut might be of value to you, but I fear that binary representation of floating point numbers is machine dependent.

    So first you have to find out on what architecture the file was written (specifically if it was big endian or little endian), nad if your scripts runs on the same kind of machine.

[reply]
Re: HEX to floating point
by kabeldag (Hermit) on Jun 16, 2008 at 12:01 UTC
    pack() should do it. If I'm thinking correctly.
[reply]
Re: HEX to floating point
by syphilis (Archbishop) on Jun 16, 2008 at 12:18 UTC
    Hi Spooky,
    I can see how 0000 0034 translates to 52.
    I can see how 0000 31b1 translates to 12721.
    I can see how 0191 translates to 1 145.
    But I can't see how 403b 8811 bb13 66e4 is supposed to translate to 27.53152055.

    Any clues regarding the rules governing that last conversion ?

    Cheers,
    Rob
    Update: I see that almut has taken the time to demonstrate the obviousness of the "rules" about which I asked.
[reply]

Back to Seekers of Perl Wisdom