Re: Why does Perl get slower when building a larger hash? (Not due to the memory swapping)

[kcott]

in reply to Why does Perl get slower when building a larger hash? (Not due to the memory swapping)

G'day chialingh,

Welcome to the monastery.

The first thing I'd suggest doing is adding use warnings; (warnings) after use strict; and fix up problems that are reported. There may be others, but a couple that grabbed my attention were close IN; (a filehandle that is never opened) and my %hash1; ... my %hash1 ... (declared twice in same scope - ditto for %hash2).

One place where I believe you're doing unnecessary processing is:

my %file_genes = %$file_gene_ref;
...
  %hash1 = %{$file_genes{$m1}};
  %hash2 = %{$file_genes{$m2}};
...
    $intersection{$_} = $hash1{$_} if exists $hash2{$_};
...

Rather than performing all those hash dereferences, I believe you could simply write something like:

    $intersection{$_} = $file_gene_ref->{$m1}{$_}
        if exists $file_gene_ref->{$m2}{$_};

You'd need to handle other references to %hash1 and %hash2 in a similar fashion. At the very least, I don't believe you need to create the intermediate hash %file_genes: you could just create %hash1 and %hash2 directly, like this:

  %hash1 = %{$file_gene_ref->{$m1}};
  %hash2 = %{$file_gene_ref->{$m2}};

If that doesn't help, you'll need to profile your code. Devel::NYTProf may be a good place to start; other modules listed under CPAN - Development Support might be useful.

[chialingh]

Hi Ken,

Thank you so much! Your solution works! :D

I'm very curious about the reason behind this. Is it due to the memory usage?

Chialingh