|Perl: the Markov chain saw|
Re: Modify values of tied, split lines in a fileby BrowserUk (Pope)
|on Oct 22, 2012 at 19:31 UTC||Need Help??|
I'm using Tie::File to read the input file line-by-line as an array. I understand that I can also use this method to modify records in a file, which would cut out the seemingly wasteful step of making a new file...
Tie::File is horribly inefficient when modifying large files, in-place. Think about what it has to do when you modify a line.
Say your file consists of:
And you decide to modify line 2 so that it looks like this:
In order to accommodate that single extra character, every one of the 10,000,001 lines following it, will have to be read, and then re-written.
And then you add or delete, a character in line 3 and the same process has to be repeated again.
Of course, Tie::File is more intelligent than that and it goes to great lengths to buffer changed records in memory and defer the re-writing of the file until it has accumulated a bunch of changes. But that caching of changes does not come for free. It needs substantial memory and substantial cpu to be effective.
And if you need to make changes to a substantial proportion of the lines, in the end, even with the caching, lots more data gets read and re-written for each change than is the case when you read from one file, make a change, and write to another, in a simply linear flow.
(*The only exception I would make is for fixed record-length files, where changes to one line do not require all the following lines to be rewritten to accommodate each change.)