Re: Compare two large files and extract a matching row...

[mzedeler]

Here is an approach that should work:

1. Read file 1, pulling field 2 out and storing it in a hash for fast lookup.

while(<FILE1>) {
    my @fields = split /|/;
    $lookup{$fields[1]} = 1;
}

2. Read file 2 line by line and put the line in your output file if field 5 is in the %lookup hash.

By using this approach, you will never need to have more than one line from each file in memory at any time, and the ids from file 2. This should be both fast and memory efficient.