Re: Memory usage while tallying instances of lines in a .txt file

[choroba]

Help us to help you! Next time, please post code and data we can easily test. See my code below for an example - it should work right ahead.

To print 0 instead of undef, you can use the defined-or // operator. It needs Perl 5.10, otherwise you have to be more verbose (defined $_ ? $_ : 0).

#!/usr/bin/env perl
use warnings;
use strict;
use feature qw{ say };

use List::Util qw{ first };

my $IN = *DATA{IO};
my %hits;

my @headers = split ' ', <$IN>;
my $index = first { $headers[$_] eq 'Strand' } 0 .. $#headers;
while (<$IN>) {
    chomp;
    my @F = split ' ';
    $hits{ $F[ $index + 1 ] }{ $F[ $index + 2 ] }{ $F[$index] }++;
}

for my $key (keys %hits) {
    for my $inner_key (keys %{ $hits{$key} }) {
        say join "\t", $key, $inner_key,
            map $_ // 0, @{ $hits{$key}{$inner_key} }{qw{ c w }};
    }
}

__DATA__
Strand
c    8    336158    75    75M    74
c    12    828707    74    74M    73
w    10    528559    74    74M    0
c    15    267766    74    74M    73
c    12    828707    74    74M    73
c    14    491797    74    74M    73

Have you noticed I used $_ only in map and first? Both chomp and split work with it by default. If you feel the need to explicitly type the argument, name the variable.

[TJCooper]

Thanks. This saves about 300-350MB of RAM! I was not at all aware of the // operator so decided not to rely on autovivification and opted for the approach shown in the OP.

[BrowserUk]

The following code produces identical results to choroba's code but uses less than 1/4 of the memory (180MB vs 795MB for my test dataset) and runs more quickly:

#! perl -slw
use strict;
use List::Util qw[ first ];

my @headers = split ' ', scalar <>;
my $f = first { $headers[$_] eq 'Strand' } 0 .. $#headers;

my( $cCounts, $wCounts, $n, %index ) = ( '', '', 0 );

while( <> ) {
    chomp;
    my @F = split ' ';
    my $index = $index{ $F[ $f+1 ] }{ $F[ $f + 2 ] } //= $n++;
    ++vec( $F[ $f ] eq 'w' ? $wCounts : $cCounts, $index, 8 );
}

while( my( $key, $subhash ) = each %index ) {
    while( my( $subkey, $index ) = each %{ $subhash } ) {
        print join "\t", $key, $subkey, vec( $cCounts, $index, 8 ), ve
+c( $wCounts, $index, 8 );
    }
}
__END__
1177246.pl 1177246.dat > 1177246.out

It assumes no count will be greater than 256. If that's too small, change the three 8s to 16s for a small increase in memory consumption.

[RonW]

It assumes no count will be greater than 256. If that's too small, change the three 8s to 16s

Actually, the 8s give a max count of 255 (0b00000000 .. 0b11111111, where 0b11111111 == 255)

16s will give you counts up to 65535 (0b1111111111111111).