Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
I'm using Tie::File to read the input file line-by-line as an array. I understand that I can also use this method to modify records in a file, which would cut out the seemingly wasteful step of making a new file...

Don't! (*)

Tie::File is horribly inefficient when modifying large files, in-place. Think about what it has to do when you modify a line.

Say your file consists of:

1, 2, 3 4, 5, 6 7, 8, 9 ...{10 million more lines here}

And you decide to modify line 2 so that it looks like this:

1, 2, 3 4, 55, 6 7, 8, 9 ...{10 million more lines here}

In order to accommodate that single extra character, every one of the 10,000,001 lines following it, will have to be read, and then re-written.

And then you add or delete, a character in line 3 and the same process has to be repeated again.

Of course, Tie::File is more intelligent than that and it goes to great lengths to buffer changed records in memory and defer the re-writing of the file until it has accumulated a bunch of changes. But that caching of changes does not come for free. It needs substantial memory and substantial cpu to be effective.

And if you need to make changes to a substantial proportion of the lines, in the end, even with the caching, lots more data gets read and re-written for each change than is the case when you read from one file, make a change, and write to another, in a simply linear flow.

(*The only exception I would make is for fixed record-length files, where changes to one line do not require all the following lines to be rewritten to accommodate each change.)


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong

mod://Tie::File

In reply to Re: Modify values of tied, split lines in a file by BrowserUk
in thread Modify values of tied, split lines in a file by glemley8

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (6)
As of 2024-04-18 20:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found