Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

unpack() removing data

by shnatko (Initiate)
on Jan 10, 2013 at 21:58 UTC ( #1012748=perlquestion: print w/ replies, xml ) Need Help??
shnatko has asked for the wisdom of the Perl Monks concerning the following question:

I'm running into a strange issue with the unpack() function. I have a command which produces output in binary format which I'm trying to unpack into a hex string for parsing, printing, etc.. It appears that unpack() is removing a byte from the binary data when the hex string is created and I can't figure out why.

I put a small test script together to play around with it a bit and can work around it by writing the binary data to a file first then reading it back in but it seems quite bizarre that doing it the 2 ways would yield different results

use strict; my $data = qx{sg_logs --page=0x34,1 pd1 -H -r > temp.bin}; $data = qx{sg_logs --page=0x34,1 pd1 -H -r }; $data = unpack("H*", $data); open OUTFILE1, ">", "outfile1.txt" or die $!; print "unpacked1: ".$data."\n"; print OUTFILE1 $data; close OUTFILE1; open FILE, "temp.bin" or die $!; binmode FILE; open OUTFILE2, ">", "outfile2.txt" or die $!; while (<FILE>) { print "unpacked2: ".unpack('H*', $_ )."\n"; print OUTFILE2 unpack('H*', $_); } close (FILE); close (OUTFILE2);

When I inspect the 2 output files created, outfile1.txt is always missing a byte in the same location ( 55th byte ) vs. outfile2.txt and of course, it's a byte I'm particularly interested in when parsing the results :) Comparing the 2 file sizes also shows the 2byte difference. Any help would be greatly appreciated!

Comment on unpack() removing data
Download Code
Re: unpack() removing data
by BrowserUk (Pope) on Jan 10, 2013 at 22:17 UTC
    open FILE, "temp.bin" or die $!; binmode FILE; open OUTFILE2, ">", "outfile2.txt" or die $!; while (<FILE>) {

    Reading a binary file using realine (aka. the <> operator) is wrong. Depending on your OS and the current setting of $/, weird things can happen when a binary value (integer; short, long,long; signed, unsigned) happens to contain a byte that looks like the delimiter.

    You should use read or sysread or set $/ = undef and slurp, or set $/ = \nnn; and read fixed length chunks.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      The strange part is that gives me the results I'm expecting ( byte not stripped out ). It's NOT reading the binary results from the temp.bin file which is causing me problems.

        On my system (Fedora 17, perl v5.14.3) it strips out hex 0D (CR) in both cases. Handling of line terminators is documented as being system-dependent, though. Without knowing exactly which character being deleted (and the characters before and after it) and what flavor of system you are running on it's hard to guess what the problem might be. You might try this and see if the two dumps are different; if they are you can start looking at sg_logs rather than at your perl script...

        sg_logs --page=0x34,1 pd1 -H -r > temp.bin od -x temp.bin | head sg_logs --page=0x34,1 pd1 -H -r | od -x | head

        Without sample data (ie. the output from sglogs which I have no knowledge of), I cannot guess further.

        But I bet the root cause is the reading or writing of binary values using functions designed for delimited text.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: unpack() removing data
by shnatko (Initiate) on Jan 11, 2013 at 19:51 UTC
    the problem appears to be in the
    $data = qx{sg_logs --page=0x34,1 pd1 -H -r };
    line where the 0x0D byte which is part of the sg_logs result is getting stripped when sent to $data with the '=' operator is there a way to ensure the result from the qx{} is sent to $data as binary data without any interference? something like binmode but for '=' instead of for file handles?

      Instead of using qx//, use a piped open whereby you can binmode the handle before importing the data:

      open PIPE, '-|', 'sg_logs --page=0x34,1 pd1 -H -r' or die $!; binmode PIPE; my $data = do{ local $/; <PIPE> }; close PIPE;

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        thank you! this is exactly what I needed. looks to be working correctly now.

        thanks again

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1012748]
Approved by toolic
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (14)
As of 2014-10-30 19:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (208 votes), past polls