Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Memory use when reading a text file

by abingham (Initiate)
on Jul 24, 2002 at 11:44 UTC ( #184781=perlquestion: print w/ replies, xml ) Need Help??
abingham has asked for the wisdom of the Perl Monks concerning the following question:

Below is my code snippet...

...simple job, open a text log file, read through it line by line and then close it.

I am actually trying to extract statistics from our mail logs, so the final code will count up the number of messages to from each address and store this in a hash to be output to CSV file at the end. However, that's getting beyond myself.

The problem is when I run this with a 46Mb file, my WinXp workstation uses up all 1GB of RAM + Swap and goes ape. This seems alot of memory usage to just read through a file, even if the machine caches the entire file in memory. With this code, I am not storing any of the data or manipulating it in any way!

Any ideas from the Perl Monks?

Is there a more efficient way of reading through files?

# process.pl # andy bingham # 14 April 2002 # open(INP, "log.txt") or die "cannot open input"; foreach $line (<INP>) { # read in each line of INP #processing code goes here } close(INP); # # DONE #

Comment on Memory use when reading a text file
Download Code
Replies are listed 'Best First'.
Re: Memory use when reading a text file
by Abigail-II (Bishop) on Jul 24, 2002 at 11:49 UTC
    You are using <INP> in list context. That will make Perl slurp in the entire file, and turn it into a list of lines.

    The standard idiom of processing a file line by line is:

    while (<IN>) { # Processing goes here; the current line is in $_ }
    That should cut your memory usage as it doesn't read in the entire file in memory (unless there's just one line in the file...)

    Abigail

Re: Memory use when reading a text file
by ichimunki (Priest) on Jul 24, 2002 at 11:51 UTC
    This is related to your use of for(each), which tries to store the whole file in a temporary array in memory.

    Try:

    while( my $line = <INP> ){ ... }
    instead. This will simply copy each line into memory as needed.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://184781]
Approved by broquaint
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (14)
As of 2015-07-30 12:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (271 votes), past polls