Re^4: Memory Efficient Alternatives to Hash of Array

[tilly]

From the OP's text and code I thought that the OP wanted to know all of the possible values for the second field for each possible value of the first field. Given that the first field repeats, this requires an array.

[neversaint]

You are exactly right, tilly.

[BrowserUk]

I stand corrected.

However, you're still better off using an external sort, as it allows you to gather the multiple values for each key together without loading the entire dataset into memory. Using a fairly simple loop like this:

#! perl 
use strict;

my( $key, @array ) = split "\t", <>;
while( <> ) {
    chomp;
    my( $newKey, $value ) = split "\t";
    if( $newKey eq $key ) {
        push @array, $value;
        next;
    }
    else {
        # Process @array for $key      
        
        #...
        
        ## Remember the newKey
        $key = $nextKey;
        ## And the reset the array
        @array = $value;
    }
}

And a command line like:

sort < unsortedFile | perl theScriptAbove

Or just sort the file and then feed it to the script as separate steps:

sort < unsortedFile > sortedFile
perl theScriptAbove sortedFile

[tilly]

*ahem* My first answer had the same idea, but with one less bug.