Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Reading binary file in perl having records of different length

by andal (Friar)
on Jun 17, 2014 at 07:10 UTC ( #1090106=note: print w/ replies, xml ) Need Help??


in reply to Reading binary file in perl having records of different length

In general, your approach is fine. Perl function "read" uses streams which in turn use data caching. So, it is fast to read even by 2 bytes. If your files are only few megabytes large, then you may read them completely into string but I don't think it will improve speed, because in my tests, stream was still obtaining data in the same chunks size, so number of system calls didn't change.

One comment on your use of "pack/unpack". Somehow you overuse it. For example, when searching for eye catcher just do "next unless $buffer eq '==';". When converting binary length just do "my $length = unpack('s', $buffer);".

One more thing. You don't check the return value of "read". Especially when you read "$length - 4" bytes. The file might be corrupted and you'll never get desired number of bytes. Plus, you say, your records may contain incorrect length, then what would be you strategy for recovery in this situation? Potentially the length may point to the middle of the next record.


Comment on Re: Reading binary file in perl having records of different length
Re^2: Reading binary file in perl having records of different length
by jaypal (Beadle) on Jun 17, 2014 at 15:40 UTC

    Thanks andal. All great suggestions. The binary length suggestion didn't seem to work in my case. I want to capture highest nibble first.

    For example, the binary length in hex is 0x03 0x50 so I need to read next 848 bytes. However if I do unpack('s', $buffer) it returns 20483 bytes (reading the binary data as 0x50 0x03).

      See pack patterns "n" and "v" instead, or the "<" and ">" modifiers if you really want signed interpretation.

      $ perl -le 'print unpack("v","\x03\x50")' 20483 $ perl -le 'print unpack("n","\x03\x50")' 848 $ perl -le 'print unpack("s<","\x03\x50")' 20483 $ perl -le 'print unpack("s>","\x03\x50")' 848

        Awesome!! The use of "n" pattern worked great.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1090106]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2014-12-19 06:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (71 votes), past polls