Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Slurping file into array VS Process file line-by-line

by monkfan (Curate)
on Oct 27, 2004 at 07:15 UTC ( #402915=perlquestion: print w/replies, xml ) Need Help??

monkfan has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have a question, that may seem trivial to you, I hope you don't mind to look at it.
This is about treating file from input argument.

I wonder:
1. When we should choose to slurp a file into array and processing it, (see Example 1) or
2. Processing it line by line (see Example 2).
3. What are the pro and cons of using either technique.

Example 1:
$filename = $ARGV[0]; #corrected from @ARGV[0] as suggested by eyepop +slikeamosquito open (FILE,"$filename") || die "Can't Open $filename: $!\n"; @LINES=<FILE>; #slurping into array foreach my $line(@LINES){ #process something here }
Example 2:
$filename = $ARGV[0]; open (FILE,"$filename") || die "Can't Open $filename: $!\n"; while<FILE>{ #dealing with it per-lines #process something here }
Regards,
Edward

Replies are listed 'Best First'.
Re: Slurping file into array VS Process file line-by-line
by Corion (Pope) on Oct 27, 2004 at 07:21 UTC

    The first example never makes sense in the given form. If you are processing stuff sequentially, from beginning to end, it makes sense to use the file iterator instead of slurping it into memory, as by that way, you will not use as much system memory and perls buffering will handle the reading from the file in a relatively efficient manner.

    As soon as you need to go back and forth between lines, and jump backwards in the files, then the first approach of reading the whole file into memory makes the algorithm much easier to write, at the cost of some system memory. If system memory is scarce and you are able to trade a bit of time against memory, take a look at Tie::File, which lets you access a file on disk just like an array in memory.

Re: Slurping file into array VS Process file line-by-line
by jarich (Curate) on Oct 27, 2004 at 09:26 UTC
    Corion and eyespoplikeamosquito have discussed your specific code examples, so I'll discuss the general question:

    1. When we should choose to slurp a file into array and processing it
    2. Processing it line by line
    3. What are the pro and cons of using either technique

    The short answer is that you should always process files line by line unless you can't otherwise avoid it.

    Times when you might not be able to avoid it are when you wish to do some sort of processing on the whole file, for example, sorting all the lines (although there's usually better tools for this) or picking out a random quote (although there are better tools for this too).

    If you don't have to be able to access all the lines of a file in a semi-random order then you should always process it line by line.

    Why? Because one day someone will feed your program a very large file. Perhaps one which is 10 Gb big. If you process that line by line your memory footprint will be comparatively small, if you slurp it all into memory, then your memory footprint will be *at least* as big as the file, sometimes almost twice that, and your machine might come grinding to a halt.

    The pro of processing a file line by line is that your program stays fast and usable. The con is that you can only refer to lines you've already seen (if you've stored them) and the current lines. Lines further into the file are much more difficult to access. Fortunately you don't usually have to.

    The pro of slurping a whole file into memory is that you can access any line of it easily. The con is that you might need a lot of memory, and it can really slow your machine down.

    Hope that helps.

    jarich

      Times when you might not be able to avoid it are when you wish to do some sort of processing on the whole file / ... / picking out a random quote

      Note that you don't have to have knowledge of the whole list to randomly select an element in it. In the particular example of lines from files, see perlfaq5, "How do I select a random line from a file?".

      ihb

      See perltoc if you don't know which perldoc to read!
      Read argumentation in its context!

Re: Slurping file into array VS Process file line-by-line
by eyepopslikeamosquito (Chancellor) on Oct 27, 2004 at 08:09 UTC

    Your general coding style is reminiscent of Perl 4. So I hope you don't mind if I make a few general style suggestions.

    $filename = @ARGV[0];

    I think you meant to type $ARGV[0]. Writing @ARGV[0] is a slice and is almost never what you want.

    open (FILE,"$filename") || die "Can't Open $filename: $!\n";

    Prefer the low precedence or to ||. Also, you don't need the quotes around $filename. Finally, unless you are stuck with an ancient perl, employ the modern three argument form of open with a lexical file handle (see example code below).

    Also, don't forget the option of slurping into a string rather than an array. Quite often, it's more convenient to process the file contents with regexes on a string, rather than line by line from an array. For example.

    use strict; my $filename = shift; open(my $fh, '<', $filename) or die "Can't open $filename: $!"; my $text = do { local $/; <$fh> }; # slurping into a string # do something with $text here

    Update: for more information on file slurping idioms, see Load file into a scalar without File::Slurp, especially merlyn's response.

Re: Slurping file into array VS Process file line-by-line
by TedPride (Priest) on Oct 27, 2004 at 11:13 UTC
    Read is also a possibility if you're trying to find a specific string in the file, and don't want to have to do the same search on many different lines of a relatively small file (few MB or less). Read gives slightly better performance than either array / line based option.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://402915]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (3)
As of 2019-08-24 22:02 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?