Re: Why does Perl get slower when building a larger hash? (Not due to the memory swapping)

in reply to Why does Perl get slower when building a larger hash? (Not due to the memory swapping)

Welcome to the monastery.

The first thing I'd suggest doing is adding use warnings; (warnings) after use strict; and fix up problems that are reported. There may be others, but a couple that grabbed my attention were close IN; (a filehandle that is never opened) and my %hash1; ... my %hash1 ... (declared twice in same scope - ditto for %hash2).

One place where I believe you're doing unnecessary processing is:

my %file_genes = %$file_gene_ref;
...
  %hash1 = %{$file_genes{$m1}};
  %hash2 = %{$file_genes{$m2}};
...
    $intersection{$_} = $hash1{$_} if exists $hash2{$_};
...
[download]

Rather than performing all those hash dereferences, I believe you could simply write something like:

    $intersection{$_} = $file_gene_ref->{$m1}{$_}
        if exists $file_gene_ref->{$m2}{$_};
[download]

You'd need to handle other references to %hash1 and %hash2 in a similar fashion. At the very least, I don't believe you need to create the intermediate hash %file_genes: you could just create %hash1 and %hash2 directly, like this:

  %hash1 = %{$file_gene_ref->{$m1}};
  %hash2 = %{$file_gene_ref->{$m2}};
[download]

If that doesn't help, you'll need to profile your code. Devel::NYTProf may be a good place to start; other modules listed under CPAN - Development Support might be useful.

-- Ken

Comment on Re: Why does Perl get slower when building a larger hash? (Not due to the memory swapping) Select or Download Code

Replies are listed 'Best First'.
Re^2: Why does Perl get slower when building a larger hash? (Not due to the memory swapping) by chialingh (Initiate) on Mar 03, 2013 at 20:44 UTC
Hi Ken, Thank you so much! Your solution works! :D I'm very curious about the reason behind this. Is it due to the memory usage? Chialingh	[reply]

In Section Seekers of Perl Wisdom