Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

If you want to slurp the entire file, don't read it line by line and then build up a giant $printout string. That will result (internally) in lots of copying and reallocating of larger memory chunks as the string grows. ...Perl's pretty smart, but you could almost look at that technique as an optimized version of Schlemiel the Painter's Algorithm. As you append, Perl has to allocate a larger chuncks of memory, and move the entire string to the new, larger locations.

Better to just slurp it in one step using one of the following:

my $string = do{ open my $fh, '<', 'filename' or die $!; local $/ = undef; <$fh>; };

...or...

use File::Slurp; my $text = read_file('filename');

...or...

use File::Slurp; my @lines = read_file('filename');

Any of the above ought to be relatively efficient, though the File::Slurp methods are easier on the eyes. But if reading the file line by line (your existing approach) is too slow, slurping is only going to yield incremental savings, at best. Either way, you've got to touch the entire file. Line-by-line approaches tend to work out well, and scale well.

You're probably already aware of this, but if the file size is large enough to swamp physical memory, slurping is not a good approach.


Dave


In reply to Re: printout of large files. loop vs. flush at once by davido
in thread printout of large files. loop vs. flush at once by ISAI student

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2024-04-19 14:17 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found