Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re^2: Improving performance of checksum calculation

by Crackers2 (Parson)
on May 30, 2009 at 16:36 UTC ( #767046=note: print w/replies, xml ) Need Help??


in reply to Re: Improving performance of checksum calculation
in thread Improving performance of checksum calculation

Interesting. Here's the results of applying some of this to the real code. My real file is about 200MB.

The full code also has a few more things; it gets a list of filehandles to chain together, and a maximum length to read in total; this complicates things slightly but doesn't seem to affect performance above noise levels.

## Original code: 98.50user 0.18system 1:38.68elapsed 100%CPU ## With ikegami's change: 85.84user 0.24system 1:26.15elapsed 99%CPU ## Using 8k blocksize for reading 24.62user 0.25system 0:24.87elapsed 100%CPU ## using 32k blocksize 26.74user 0.37system 0:27.16elapsed 99%CPU

Going higher than 8K blocks doesn't seem to help much on my system; the extra time taken with 32K blocks is probably noise and would smoothen out over multiple runs.

Since the actual input can be a stream (in this case usually the output of some other program piped to mine), I couldn't move the "multiple of 4 bytes" check out of the loop; but since it's no longer in the tightest loop that doesn't appear to matter much either.

And for completeness, here's my time using your C code converted to xs:

0.38user 0.40system 0:00.95elapsed 82%CPU

One interesting thing at this point is that we've dropped CPU usage from 100% down to about 80%, indicating that at this point reading the file might be the bottleneck.

Of course, that's all relative; after going from 1:38 or 0:00.95 any remaining optimization isn't of any practical purpose for me.

Thanks to both you and Ikegami for showing me I discarded the C-in-Perl too soon. While the end result is a bit more annoying to use, the performance gains are certainly worth it.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://767046]
help
Chatterbox?
[davido]: Using the -i switch causes Perl to rename the input file, read from it, and write to a file using the original file's name. If there's an extension provided, as in -i.bak, it's easy to see where the input file is. Where is the input file temporarily....
[davido]: placed if there is no extension provided to the -i switch?
[davido]: Nevermind, found the answer.
[davido]: If no extension is supplied, and your system supports it, the original file is kept open without a name while the output is redirected to a new file with the original filename. When perl exits, cleanly or not, the original file is unlinked.
[haukex]: doc says "If no extension is supplied, and your system supports it, the original file is kept open without a name ..."

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (14)
As of 2017-09-22 14:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    During the recent solar eclipse, I:









    Results (264 votes). Check out past polls.

    Notices?