Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re: Slurping file into array VS Process file line-by-line

by jarich (Curate)
on Oct 27, 2004 at 09:26 UTC ( #402937=note: print w/ replies, xml ) Need Help??


in reply to Slurping file into array VS Process file line-by-line

Corion and eyespoplikeamosquito have discussed your specific code examples, so I'll discuss the general question:

  1. When we should choose to slurp a file into array and processing it
  2. Processing it line by line
  3. What are the pro and cons of using either technique

The short answer is that you should always process files line by line unless you can't otherwise avoid it.

Times when you might not be able to avoid it are when you wish to do some sort of processing on the whole file, for example, sorting all the lines (although there's usually better tools for this) or picking out a random quote (although there are better tools for this too).

If you don't have to be able to access all the lines of a file in a semi-random order then you should always process it line by line.

Why? Because one day someone will feed your program a very large file. Perhaps one which is 10 Gb big. If you process that line by line your memory footprint will be comparatively small, if you slurp it all into memory, then your memory footprint will be *at least* as big as the file, sometimes almost twice that, and your machine might come grinding to a halt.

The pro of processing a file line by line is that your program stays fast and usable. The con is that you can only refer to lines you've already seen (if you've stored them) and the current lines. Lines further into the file are much more difficult to access. Fortunately you don't usually have to.

The pro of slurping a whole file into memory is that you can access any line of it easily. The con is that you might need a lot of memory, and it can really slow your machine down.

Hope that helps.

jarich


Comment on Re: Slurping file into array VS Process file line-by-line
Re^2: Slurping file into array VS Process file line-by-line
by ihb (Deacon) on Oct 27, 2004 at 10:19 UTC

    Times when you might not be able to avoid it are when you wish to do some sort of processing on the whole file / ... / picking out a random quote

    Note that you don't have to have knowledge of the whole list to randomly select an element in it. In the particular example of lines from files, see perlfaq5, "How do I select a random line from a file?".

    ihb

    See perltoc if you don't know which perldoc to read!
    Read argumentation in its context!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://402937]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2014-11-01 07:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (228 votes), past polls