http://www.perlmonks.org?node_id=1012748

shnatko has asked for the wisdom of the Perl Monks concerning the following question:

I'm running into a strange issue with the unpack() function. I have a command which produces output in binary format which I'm trying to unpack into a hex string for parsing, printing, etc.. It appears that unpack() is removing a byte from the binary data when the hex string is created and I can't figure out why.

I put a small test script together to play around with it a bit and can work around it by writing the binary data to a file first then reading it back in but it seems quite bizarre that doing it the 2 ways would yield different results 

 use strict;

my $data = qx{sg_logs --page=0x34,1 pd1 -H -r > temp.bin};     
$data = qx{sg_logs --page=0x34,1 pd1 -H -r };                 
$data = unpack("H*", $data);

open OUTFILE1, ">", "outfile1.txt" or die $!;
print "unpacked1: ".$data."\n";    
print OUTFILE1 $data;
close OUTFILE1;
        
open FILE, "temp.bin" or die $!;
binmode FILE;

open OUTFILE2, ">", "outfile2.txt" or die $!;

while (<FILE>) {
    print "unpacked2: ".unpack('H*', $_ )."\n";
    print OUTFILE2 unpack('H*', $_);
}

close (FILE);
close (OUTFILE2);

[download]

When I inspect the 2 output files created, outfile1.txt is always missing a byte in the same location ( 55th byte ) vs. outfile2.txt and of course, it's a byte I'm particularly interested in when parsing the results :) Comparing the 2 file sizes also shows the 2byte difference. Any help would be greatly appreciated!

Replies are listed 'Best First'.
Re: unpack() removing data
by BrowserUk (Patriarch) on Jan 10, 2013 at 22:17 UTC 
    open FILE, "temp.bin" or die $!;
binmode FILE;

open OUTFILE2, ">", "outfile2.txt" or die $!;

while (<FILE>) {

    [download]

    Reading a binary file using realine (aka. the <> operator) is wrong. Depending on your OS and the current setting of $/, weird things can happen when a binary value (integer; short, long,long; signed, unsigned) happens to contain a byte that looks like the delimiter.

    You should use read or sysread or set $/ = undef and slurp, or set $/ = \nnn; and read fixed length chunks.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      The strange part is that gives me the results I'm expecting ( byte not stripped out ). It's NOT reading the binary results from the temp.bin file which is causing me problems.
[reply]

        Without sample data (ie. the output from sglogs which I have no knowledge of), I cannot guess further.

        But I bet the root cause is the reading or writing of binary values using functions designed for delimited text.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        On my system (Fedora 17, perl v5.14.3) it strips out hex 0D (CR) in both cases. Handling of line terminators is documented as being system-dependent, though. Without knowing exactly which character being deleted (and the characters before and after it) and what flavor of system you are running on it's hard to guess what the problem might be. You might try this and see if the two dumps are different; if they are you can start looking at sg_logs rather than at your perl script... 

        sg_logs --page=0x34,1 pd1 -H -r > temp.bin
od -x temp.bin | head

sg_logs --page=0x34,1 pd1 -H -r | od -x | head

        [download]
[reply]
[d/l]
Re: unpack() removing data
by shnatko (Initiate) on Jan 11, 2013 at 19:51 UTC
    the problem appears to be in the 
     
 $data = qx{sg_logs --page=0x34,1 pd1 -H -r };

    [download]
    line where the 0x0D byte which is part of the sg_logs result is getting stripped when sent to $data with the '=' operator is there a way to ensure the result from the qx{} is sent to $data as binary data without any interference? something like binmode but for '=' instead of for file handles?
[reply]
[d/l]

      Instead of using qx//, use a piped open whereby you can binmode the handle before importing the data:

      open PIPE, '-|', 'sg_logs --page=0x34,1 pd1 -H -r' or die $!;
binmode PIPE;
my $data = do{ local $/; <PIPE> };
close PIPE;

      [download]
      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        thank you! this is exactly what I needed. looks to be working correctly now.

        thanks again

[reply]

Back to Seekers of Perl Wisdom