Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

Re^3: Efficiently processing a file

by rkrieger (Friar)
on Jan 26, 2010 at 07:51 UTC ( #819656=note: print w/replies, xml ) Need Help??

in reply to Re^2: Efficiently processing a file
in thread Efficiently processing a file

The foreach loop is hungry in that it stores all lines in memory. A while loop can avoid that by merely nibbling at the cake until its gone. Without knowing the actual details you face, I assume the speed issue is one of reading the file rather than processing individual lines. That's the key bit others pointed out.

While I do not know what your count_bases() function intends to do, splitting the reading and processing steps may help you. I assume you want to do regex matching or something else that can be done on individual lines.

while (my $line = <>) { foreach my $count (20, 30) { count_bases($line, $count); } }

The foreach loop makes it easy to extend the particular processing steps on individual lines (e.g. through a dispatch table with code references). The while loop simply keeps running as long as there is input, not trying to store it all in memory at once.

Code references (a reference to 'code') are very useful for dispatch tables: they allow you to easily parametrize behaviour of your program. You can store them in arrays or hashes and later on loop over those arrays to ensure all (or only specific) actions are taken. Higher Order Perl by Mark Jason Dominus has a chapter that I find quite instructive.

A (particularly useless) example of what I mean is below. Adding more steps is trivial: add more items to @actions (making sure the subs accept the same arguments). You could also use hashes of course and select the code to be executed based on actual input.

#!/usr/bin/perl use strict; use warnings; sub prefix_line { my $lineno = shift @_; my $line = shift @_; # Odd line? Yes No my $prefix = $lineno % 2 ? q{+} : q{o}; print qq{$prefix }; } sub print_line { my $lineno = shift @_; my $line = shift @_; # Print our actual line print qq{$line}; } my @actions = ( \&prefix_line, \&print_line, ); my $lineno = 0; while (my $line = <>) { $lineno++; foreach my $action (@actions) { $action->($lineno, $line); } }

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://819656]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (16)
As of 2017-07-20 14:13 GMT
Find Nodes?
    Voting Booth?
    I came, I saw, I ...

    Results (304 votes). Check out past polls.