Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^2: Fast parsing for cypher block chaining

by fluffyvoidwarrior (Monk)
on Feb 28, 2006 at 18:26 UTC ( #533447=note: print w/ replies, xml ) Need Help??


in reply to Re: Fast parsing for cypher block chaining
in thread Fast parsing for cypher block chaining

Because I thought read would be slower than sysread and the 8 byte buffer would result in a lot of disk thrashing compared to 64k buffer


Comment on Re^2: Fast parsing for cypher block chaining
Re^3: Fast parsing for cypher block chaining
by ikegami (Pope) on Feb 28, 2006 at 18:28 UTC
    the 8 byte buffer would result in a lot of disk thrashing compared to 64k buffer

    And there shouldn't be any disk trashing because read and write are buffered. The size of the buffer isn't 8 bytes as you claim, but rather a multiple of the size of a disc sector. The disk is not accessed every time read and write is called.

    Because I thought read would be slower than sysread

    sysread + syswrite + manual buffering in Perl
    should be slower than
    read + write + well tested buffering in C

    Note that I didn't say my way is faster, just that it might be. Benchmark to find out which is faster on your system.

      Just benchmarked a comparison between sysread/write with substr() and straight perl IO read/write if anyone is still interested.
      The results are a bit of a shock!
      Parsing a 700Mb file in 8 byte chunks took 81 seconds with the sysread method. Using perl buffered read/write it took 522 seconds. It seems sysread and handling your own buffering can produce performance gains of upto 700% - which is what I'm looking for.
      Heres my code just in case I've done anything dumb (I'm assuming using OOP IO is OK)
      $infile = new IO::File; $outfile = new IO::File; $infile->open($input_filepath); $outfile->open(">$output_filepath"); for($chunk_counter = 0; $chunk_counter < $infile_num_chunks +1; + $chunk_counter = $chunk_counter + 1){ $infile->read($buffer, 8); $outfile->write($buffer,8); }

      I originally used a while construct for loop control but then thought maybe it was slowing things down. It was. Using "while" took 522 secs. Using the counter as above took 449 secs. Either way sysread and substr() is loads faster.
        That is not benchmark code

        If it done right, it should be faster than this buggy code you show here. I say buggy since you're assuming the input file is a multiple of 8 bytes.

        Using IO::File is probably much slower than using read and write. Objects are slower than their non-object equivalent.

        If you want to use sysread and syswrite, you can use them with the solution in Re: Fast parsing for cypher block chaining, which is what you should be using.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://533447]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (14)
As of 2014-07-31 13:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (248 votes), past polls