Beefy Boxes and Bandwidth Generously Provided by pair Networks RobOMonk
Do you know where your variables are?
 
PerlMonks  

How to concatenate N binary buffers?

by mantager (Sexton)
on Nov 13, 2012 at 16:59 UTC ( #1003666=perlquestion: print w/ replies, xml ) Need Help??
mantager has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,
I'm really seeking for your wisdom, since I can't get up with an aswer from the net to this question: how do I concatenate two buffers I got from a couple of sysread()?

Here is my pseudocode:

my $bufsize = 256 * 1024; sysopen my $in1, 'somefile', O_RDONLY; sysopen my $in2, 'someother', O_RDONLY; sysopen my $out, 'outputfile', O_WRONLY|O_CREAT|O_TRUNC; binmode($in1); # Is this necessary? binmode($in2); binmode($out); while (sysread($in1, my $buf1, $bufsize) && sysread($in2, my $buf2, $bufsize)) { my $data = <CONCAT $buf1 & $buf2>; # HOW? syswrite $out, $data, 2 * $bufsize; }

Anyone can help me, please?
Thanks in advance.

Comment on How to concatenate N binary buffers?
Download Code
Re: How to concatenate N binary buffers?
by choroba (Abbot) on Nov 13, 2012 at 17:04 UTC
    . is the concatenation operator. Rather than writing 2 * $bufsize, remember how much you have read:
    my $size1 = sysread $in1, my $buf1, $bufsize;
    لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      Thanks,
      I was wondering if "." could change in some way the content of the two buffers. Since it's string concatenation operator, I was worried that some sort of "stringification" of the two buffers could change them some way (this is why I asked if there's a way to concat two binary buffers). I am still not sure about this.

      Regarding the size of the read data: I am actually trying to put back together some stripes from a dead raid array, so if I am not able to read exactly $bufsize bytes I'd better die away asap :)

      Thanks a lot!

        I was wondering if "." could change in some way the content of the two buffers. Since it's string concatenation operator, I was worried that some sort of "stringification" of the two buffers could change them some way (this is why I asked if there's a way to concat two binary buffers). I am still not sure about this.

        Perl's strings are "8-bit-clean". You can store binary data in them, and you can use all string operators on them, even if you consider your data binary. Stringification does not happen here, because for Perl, the data is already a string.

        Regarding the size of the read data: I am actually trying to put back together some stripes from a dead raid array, so if I am not able to read exactly $bufsize bytes I'd better die away asap :)

        Consider using dd conv=noerror, dd_rescue or ddrescue instead of Perl. If you care about your data, don't touch it and pay experts to rescue the data.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re: How to concatenate N binary buffers?
by RichardK (Priest) on Nov 13, 2012 at 17:07 UTC

    Why bother? why not just write each buffer separately?

    syswrite $out, $buf1, $bufsize; syswrite $out, $buf2, $bufsize;
          syswrite $out, $buf1, $bufsize;
          syswrite $out, $buf2, $bufsize;

      If a constant number  $bufsize of bytes is always written, then when fewer than  $bufsize bytes are read, typically at the end of a file, a block of junk will be written to the output file between the  $buf1 and  $buf2 blocks (Update: and also from the end of  $buf2 to the end of the output file).

      Better to follow choroba's advice and record the bytes actually read, then write just that number of bytes (but still without the need for explicit concatenation):
          syswrite $out, $buf1, $size1;
          syswrite $out, $buf2, $size2;

      It's what I am going to do, but it seemed strange to repeat the syswrite N times, one for each data chunk. I was just wandering if there's a "safe" way to concatenate data (meaning: does the "." operator change the buffers or not?) and then issue just one write (writing a larger chunk should should also optimize I/O).

      Another interesting question: is it faster to concat data and then write once, or is it faster to issue one syswrite for each buffer?
      I'm afraid I'll have no time to benchmark.

      Thanks.

        does the "." operator change the buffers or not?
        T.I.T.S. Or, Try It To See.:
        # perl -E '$s .= chr for 0 .. 31; say $s' | xxd 0000000: 0001 0203 0405 0607 0809 0a0b 0c0d 0e0f ................ 0000010: 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f ................ 0000020: 0a
        I'll have no time to benchmark
        I am afraid we will have no time to answer.
        لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
        Another interesting question: is it faster to concat data and then write once, or is it faster to issue one syswrite for each buffer? I'm afraid I'll have no time to benchmark.

        You try to read data from a broken RAID. And your main problem is performance? Sorry, I don't get it. You should be happy with every single bit you can still read. And while one can use perl for data rescue, I think you should not use perl for the job. There are better tools, and you seem to know none of them, including Perl. Pay an expert to recover your RAID. If you need your data back fast, you'll probably have to pay an extra fee.

        Questions you should ask later: Why did nobody notice that the RAID had problems before it failed? Why is there no recent, verified backup of the RAID?

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re: How to concatenate N binary buffers?
by AnomalousMonk (Monsignor) on Nov 13, 2012 at 18:15 UTC
        my $bufsize = 256 * 1024;

    I'm so paranoid, I would use a constant of some kind to represent a constant value, e.g.:
        use constant BUFFSIZE => 256 * 1024;

Re: How to concatenate N binary buffers?
by Anonymous Monk on Nov 13, 2012 at 21:53 UTC
    Just push 'em onto a list. Problem solved.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1003666]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (12)
As of 2014-04-18 18:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (471 votes), past polls