Slurping file into array VS Process file line-by-line

[monkfan]

Dear Monks,

I have a question, that may seem trivial to you, I hope you don't mind to look at it.
This is about treating file from input argument.

I wonder:
1. When we should choose to slurp a file into array and processing it, (see Example 1) or
2. Processing it line by line (see Example 2).
3. What are the pro and cons of using either technique.

Example 1:

$filename = $ARGV[0]; #corrected from @ARGV[0] as suggested by  eyepop
+slikeamosquito
open (FILE,"$filename") || die "Can't Open $filename: $!\n";
@LINES=<FILE>;   #slurping into array

foreach my $line(@LINES){ 
    #process something here
 }

Example 2:

$filename = $ARGV[0];
open (FILE,"$filename") || die "Can't Open $filename: $!\n";

while<FILE>{ #dealing with it per-lines
  #process something here
}

[Corion]

The first example never makes sense in the given form. If you are processing stuff sequentially, from beginning to end, it makes sense to use the file iterator instead of slurping it into memory, as by that way, you will not use as much system memory and perls buffering will handle the reading from the file in a relatively efficient manner.

As soon as you need to go back and forth between lines, and jump backwards in the files, then the first approach of reading the whole file into memory makes the algorithm much easier to write, at the cost of some system memory. If system memory is scarce and you are able to trade a bit of time against memory, take a look at Tie::File, which lets you access a file on disk just like an array in memory.

[jarich]

Corion and eyespoplikeamosquito have discussed your specific code examples, so I'll discuss the general question:

1. When we should choose to slurp a file into array and processing it
2. Processing it line by line
3. What are the pro and cons of using either technique

The short answer is that you should always process files line by line unless you can't otherwise avoid it.

Times when you might not be able to avoid it are when you wish to do some sort of processing on the whole file, for example, sorting all the lines (although there's usually better tools for this) or picking out a random quote (although there are better tools for this too).

If you don't have to be able to access all the lines of a file in a semi-random order then you should always process it line by line.

Why? Because one day someone will feed your program a very large file. Perhaps one which is 10 Gb big. If you process that line by line your memory footprint will be comparatively small, if you slurp it all into memory, then your memory footprint will be *at least* as big as the file, sometimes almost twice that, and your machine might come grinding to a halt.

The pro of processing a file line by line is that your program stays fast and usable. The con is that you can only refer to lines you've already seen (if you've stored them) and the current lines. Lines further into the file are much more difficult to access. Fortunately you don't usually have to.

The pro of slurping a whole file into memory is that you can access any line of it easily. The con is that you might need a lot of memory, and it can really slow your machine down.

Hope that helps.

jarich

[ihb]

Times when you might not be able to avoid it are when you wish to do some sort of processing on the whole file / ... / picking out a random quote

Note that you don't have to have knowledge of the whole list to randomly select an element in it. In the particular example of lines from files, see perlfaq5, "How do I select a random line from a file?".

ihb

[eyepopslikeamosquito]

Your general coding style is reminiscent of Perl 4. So I hope you don't mind if I make a few general style suggestions.

$filename = @ARGV[0];

I think you meant to type $ARGV[0]. Writing @ARGV[0] is a slice and is almost never what you want.

open (FILE,"$filename") || die "Can't Open $filename: $!\n";

Prefer the low precedence or to ||. Also, you don't need the quotes around $filename. Finally, unless you are stuck with an ancient perl, employ the modern three argument form of open with a lexical file handle (see example code below).

Also, don't forget the option of slurping into a string rather than an array. Quite often, it's more convenient to process the file contents with regexes on a string, rather than line by line from an array. For example.

use strict;
my $filename = shift;
open(my $fh, '<', $filename) or die "Can't open $filename: $!";
my $text = do { local $/; <$fh> };  # slurping into a string
# do something with $text here

Update: for more information on file slurping idioms, see Load file into a scalar without File::Slurp, especially merlyn's response.

[TedPride]

Read is also a possibility if you're trying to find a specific string in the file, and don't want to have to do the same search on many different lines of a relatively small file (few MB or less). Read gives slightly better performance than either array / line based option.