Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Re: memory issues

by BrowserUk (Pope)
on Jan 28, 2013 at 09:13 UTC ( #1015649=note: print w/ replies, xml ) Need Help??


in reply to memory issues

The problem is that this line:

if($ipairs{$fk}{$pair}{'pvalue'}){

Rather that just testing if that value exists, it is autovivifying (creating) that value in the nested hashes and setting it to null.

If you change that line to:

if( exists $ipairs{$fk} && exists $ipairs{$fk}{$pair} && exists $ipairs{$fk}{$pair}{'pvalue'} ){

It should prevent the runaway memory growth. As a nice side-effect, your program should run substantially faster also.

BTW. I assume you mean 32GB not 32MB?


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.


Comment on Re: memory issues
Select or Download Code
Re^2: memory issues
by asqwerty (Acolyte) on Jan 28, 2013 at 09:24 UTC
    Thanks!!!! It works fine now and, as you said, substantially faster also. This is what I did,
    foreach my $pair (@lpairs) { my $n = 0; my $z = 0; my $hl; my $pvalue = 0; my $pvt = 0; my $fk; foreach $fk (%ipairs) { if( exists $ipairs{$fk} && exists $ipairs{$fk}{$pair} && exi +sts $ipairs{$fk}{$pair}{'pvalue'}){ #if($ipairs{$fk}{$pair}{'pvalue'}){ $pvt = $ipairs{$fk}{$pair}{'pvalue'}; if($pvt){ unless($hl){ $hl = $ipairs{$fk}{$pair}{'head'}; } $n++; $z+= qnorm($ipairs{$fk}{$pair}{'pvalue'}); } } } if($n>2){ $z = $z/sqrt($n); $pvalue = pnorm($z); } if ($pvalue) { #printf "$pair -> %.4f\n", $pvalue; printf OF "$hl %.4f $n\n", $pvalue; } }

      Just a small hint to reduce the verboseness of your code:

      ($ipairs{$fk}{$dline}{'head'}, $ipairs{$fk}{$dline}{'effect'}, $ip +airs{$fk}{$dline}{'pvalue'}) = $ldata{$dline} =~ /^(.*)\s+(\d\.\d+)\s ++\d\.\d+\s+(\d\.\d+)$/;

      can be rewritten using a hash slice like this:

      ( @{ $ipairs{$fk}{$dline} }{qw/head effect pvalue/} ) = $ldata{$dl +ine} =~ /^(.*)\s+(\d\.\d+)\s+\d\.\d+\s+(\d\.\d+)$/;

      Otherwise, your code benefits from a temporary variable or two. Here I repurpose $pvt (not sure if the variable name makes sense after that):

      $pvt = $ipairs{$fk}{$pair}; if($pvt->{'pvalue'}){ unless($hl){ $hl = $pvt->{'head'}; } $n++; $z+= qnorm($pvt->{'pvalue'}); }

      (This only works because there already exists a hash reference at $ipairs{$fk}{$pair}. It would not work if you tried to say $pvt = {}, but %$pvt = () would. It's all reference magic and not really easy to explain unless you first understand pointers.)

      (Of course, the hash slice can be rewritten using a temporary variable, too. It's always a good idea to use temporary variables if it makes your code easier to understand. Triply a good idea if it reduces repetition.)

Re^2: memory issues
by asqwerty (Acolyte) on Jan 28, 2013 at 09:25 UTC
    And you are right. It is 32 GB. :-)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1015649]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2014-09-22 05:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (178 votes), past polls