http://www.perlmonks.org?node_id=1048129


in reply to Parsing large files

The predecessor-to-Perl tool for doing this sort of thing was called awk, and you can see that its programs consist entirely of a set of regular-expressions followed by what to do when one of those was encountered in the text file.

One thing to be very mindful of, though, is how much RAM you have and how much RAM is being consumed by the program over time.   In particular, all of the memory required in the processing of one file should be completely released back to Perl before processing of the next file begins.   Although Perl does not release memory back to the operating system, nevertheless you should not see that the working-set size (WSS) of the process “continues to increase endlessly” as additional files are processed.   If it does the process can start “thrashing” and you will see this when the program (and the entire computer) grinds to a halt except for your disk-drive, whose little light never goes out.