Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)

by Anonymous Monk
on Nov 14, 2012 at 14:20 UTC ( #1003811=note: print w/ replies, xml ) Need Help??


in reply to Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)

If you need to modify a flat-file, and the size is such that it can't be slurped into memory (and a 1GB file probably can be ...), it's best to process the file one record at a time writing the modified records to another file. Then, swap the file-names around. If you slurp into memory, run a performance monitor to see if the virtual-memory subsystem starts thrashing, because VM swapping activity is a serious source of "invisible" disk I/O. If you process a record at a time, memory won't be an issue but data movement (a gigabyte is read and written) will be. But a one-gigabyte file should be processed sequentially in seconds, not 30 minutes.


Comment on Re: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1003811]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (10)
As of 2015-07-02 08:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (31 votes), past polls