Out of memory error while Reading 5GB text file

[Anonymous Monk]

Hi Monks,
I am trying to read 5GB(file size) text file and search some content in that file. But I got "Out of memory" error.
Could you please provide your suggestion.

open FIN, $inputFile or die $!;
while (my $lines  = <FIN>) {
   chomp $sword;
   unless ($lines =~ /$sword/i) {
        print "$sword\n";
   }
}
close FIN;

thanks,
PerlUser.

[varian]

Normally to read a textfile in this fashion should be just fine. However my hinch is that the code example that you presented is incomplete:

while (my $lines  = <FIN>) {
   chomp $sword;

e.g. I would have expected a chomp $lines within the loop. Maybe show us some more code details?

Now if you accidently would parse a file that is not a textfile the inputline may end up being too long and on some systems you could run out of memory.

One cause for the above could be platform differences i.e. if you read on an Windows platform a file originally created on a Unix platform the textline separator might be different and your program may fail to recognise end-of-line.
Note that this problem would not be as bad if the source is Windows and the destination is Unix. In that case you just find an extra "\r" as the last character on the line after you have chomped the input.

[idle]

Some time ago I have had same problem and asked about it here - How to save memory, parsing a big file.

[prasadbabu]

Anonymous Monk

Recently a similar question was asked and you can see those replies which will solve your problem.

Take a look at how to parse large files.

Please do Super Search before asking a question, you can find many replies.

[robot_tourist]

where is $sword set? From that fragment, it looks like you're not chomp()ing anything.

[jfroebe]

Take a look at the Tie::File module:

Tie::File represents a regular text file as a Perl array. Each element in the array corresponds to a record in the file. The first line of the file is element 0 of the array; the second line is element 1, and so on.
The file is not loaded into memory, so this will work even for gigantic files.

[ikegami]

Tie::File keeps an index of every line it encounters (which is every line up to the deepest one requested). For a 5GB file, that's a giant amount of memory. The memory option does not affect the size of this index.