Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Parsing large files

by sundialsvc4 (Monsignor)
on Aug 06, 2013 at 14:27 UTC ( #1048129=note: print w/ replies, xml ) Need Help??


in reply to Parsing large files

The predecessor-to-Perl tool for doing this sort of thing was called awk, and you can see that its programs consist entirely of a set of regular-expressions followed by what to do when one of those was encountered in the text file.

One thing to be very mindful of, though, is how much RAM you have and how much RAM is being consumed by the program over time.   In particular, all of the memory required in the processing of one file should be completely released back to Perl before processing of the next file begins.   Although Perl does not release memory back to the operating system, nevertheless you should not see that the working-set size (WSS) of the process “continues to increase endlessly” as additional files are processed.   If it does the process can start “thrashing” and you will see this when the program (and the entire computer) grinds to a halt except for your disk-drive, whose little light never goes out.


Comment on Re: Parsing large files

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1048129]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (4)
As of 2014-07-29 05:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (211 votes), past polls