Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Re: Modify values of tied, split lines in a file

by BrowserUk (Pope)
on Oct 22, 2012 at 19:31 UTC ( #1000411=note: print w/replies, xml ) Need Help??

in reply to Modify values of tied, split lines in a file

I'm using Tie::File to read the input file line-by-line as an array. I understand that I can also use this method to modify records in a file, which would cut out the seemingly wasteful step of making a new file...

Don't! (*)

Tie::File is horribly inefficient when modifying large files, in-place. Think about what it has to do when you modify a line.

Say your file consists of:

1, 2, 3 4, 5, 6 7, 8, 9 ...{10 million more lines here}

And you decide to modify line 2 so that it looks like this:

1, 2, 3 4, 55, 6 7, 8, 9 ...{10 million more lines here}

In order to accommodate that single extra character, every one of the 10,000,001 lines following it, will have to be read, and then re-written.

And then you add or delete, a character in line 3 and the same process has to be repeated again.

Of course, Tie::File is more intelligent than that and it goes to great lengths to buffer changed records in memory and defer the re-writing of the file until it has accumulated a bunch of changes. But that caching of changes does not come for free. It needs substantial memory and substantial cpu to be effective.

And if you need to make changes to a substantial proportion of the lines, in the end, even with the caching, lots more data gets read and re-written for each change than is the case when you read from one file, make a change, and write to another, in a simply linear flow.

(*The only exception I would make is for fixed record-length files, where changes to one line do not require all the following lines to be rewritten to accommodate each change.)

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1000411]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (9)
As of 2017-05-25 08:46 GMT
Find Nodes?
    Voting Booth?