Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re: Parsing large files

by sundialsvc4 (Abbot)
on Aug 06, 2013 at 14:27 UTC ( #1048129=note: print w/replies, xml ) Need Help??

in reply to Parsing large files

The predecessor-to-Perl tool for doing this sort of thing was called awk, and you can see that its programs consist entirely of a set of regular-expressions followed by what to do when one of those was encountered in the text file.

One thing to be very mindful of, though, is how much RAM you have and how much RAM is being consumed by the program over time.   In particular, all of the memory required in the processing of one file should be completely released back to Perl before processing of the next file begins.   Although Perl does not release memory back to the operating system, nevertheless you should not see that the working-set size (WSS) of the process “continues to increase endlessly” as additional files are processed.   If it does the process can start “thrashing” and you will see this when the program (and the entire computer) grinds to a halt except for your disk-drive, whose little light never goes out.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1048129]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (7)
As of 2018-01-19 02:50 GMT
Find Nodes?
    Voting Booth?
    How did you see in the new year?

    Results (215 votes). Check out past polls.