Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Perl Program - Out of memory?

by jdlev (Scribe)
on Nov 15, 2013 at 19:34 UTC ( #1062777=perlquestion: print w/ replies, xml ) Need Help??
jdlev has asked for the wisdom of the Perl Monks concerning the following question:

I'm not sure I understand this. I'm running a perl program that runs through a couple billion loops looking for an optimal scenario. It shouldn't be storing anything long term? When I run the program, it gets to a certain point and seems to stop...I'll hit control-c to stop the program and it says "Out of memory!" I'm not sure how that's even possible since my computer has almost 60GB of memory installed? Does perl have any type of error reporting or log writing abilities so I can at least see when then program is crashing?
I love it when a program comes together - jdhannibal

Comment on Perl Program - Out of memory?
Re: Perl Program - Out of memory?
by BrowserUk (Pope) on Nov 15, 2013 at 19:44 UTC

    It is impossible to suggest anything or even speculate without seeing the code.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Sorry...here's the code...should of thought of that in advance I guess..
      #USAGE EXAMPLE - Max Points Replaced If Higher Value Found require 'funcs_top10.pl'; $delay = 3; $maxSalary = 100000; $programStartTime = localtime(); $qbHash = getStats("QB"); #Sorts Player Statistics Into Hashes + According To Position $rbHash = getStats("RB"); $wrHash = getStats("WR"); $teHash = getStats("TE"); $dstHash = getStats("DST"); $flHash = getFlexStats(); foreach $key(sort keys $qbHash) { $QB1i = $qbHash->{$key}->{PlayerID}; $QB1s = $qbHash->{$key}->{Salary}; $QB1p = $qbHash->{$key}->{FPPG}; $QB1pos = $qbHash->{$key}->{Position}; #print "QB1 count = $qb1count \n"; #$qb1count++; foreach $key(sort keys $qbHash) { $QB2i = $qbHash->{$key}->{PlayerID}; $QB2s = $qbHash->{$key}->{Salary}; $QB2p = $qbHash->{$key}->{FPPG}; $QB2pos = $qbHash->{$key}->{Position}; #print "--QB2 count = $qb1count \n"; #$qb2count++; if ($QB1i != $QB2i) { foreach $key(sort keys $rbHash) { $RB1i = $rbHash->{$key}->{PlayerID}; $RB1s = $rbHash->{$key}->{Salary}; $RB1p = $rbHash->{$key}->{FPPG}; $RB1pos = $rbHash->{$key}->{Position}; #print "----RB1 count = $rb1count \n"; #$rb1count++; foreach $key(sort keys $rbHash) { $RB2i = $rbHash->{$key}->{PlayerID +}; $RB2s = $rbHash->{$key}->{Salary}; $RB2p = $rbHash->{$key}->{FPPG}; $RB2pos = $rbHash->{$key}->{Positi +on}; #print "------RB2 count = $rb2coun +t \n"; #$rb2count++; if ($RB1i != $RB2i) { foreach $key(sort keys $wrHash) + { $WR1i = $wrHash->{$key}->{ +PlayerID}; $WR1s = $wrHash->{$key}->{ +Salary}; $WR1p = $wrHash->{$key}->{ +FPPG}; $WR1pos = $wrHash->{$key}- +>{Position}; #print "--------WR1 count += $wr1count \n"; # $wr1count++; foreach $key(sort keys $wr +Hash) { $WR2i = $wrHash->{ +$key}->{PlayerID}; $WR2s = $wrHash->{ +$key}->{Salary}; $WR2p = $wrHash->{ +$key}->{FPPG}; $WR2pos = $wrHash- +>{$key}->{Position}; if($WR1i != $W +R2i) { foreach $k +ey(sort keys $teHash) #TE1 Start { $T +E1i = $teHash->{$key}->{PlayerID}; $T +E1s = $teHash->{$key}->{Salary}; $T +E1p = $teHash->{$key}->{FPPG}; $T +E1pos = $teHash->{$key}->{Position}; fo +reach $key(sort keys $dstHash) #DST Start + { + $DSTi = $dstHash->{$key}->{PlayerID}; + $DSTs = $dstHash->{$key}->{Salary}; + $DSTp = $dstHash->{$key}->{FPPG}; + $DSTpos = $dstHash->{$key}->{Position}; + + foreach $key(sort keys $flHash) #FL1 Start + { + $FL1i = $flHash->{$key}->{PlayerID}; + $FL1s = $flHash->{$key}->{Salary}; + $FL1p = $flHash->{$key}->{FPPG}; + $FL1pos = $flHash->{$key}->{Position}; + if (($FL1i != $RB1i) && ($FL1i != $RB2i) && ($FL1i +!= $WR1i) && ($FL1i != $WR2i) && ($FL1i != $TE1i)) #FL1 != If Start + { + foreach $key(sort keys $flHash) #FL2 Start + { + $FL2i = $flHash->{$key}->{PlayerID} +; + $FL2s = $flHash->{$key}->{Salary}; + $FL2p = $flHash->{$key}->{FPPG}; + $FL2pos = $flHash->{$key}->{Positio +n}; + if (($FL2i != $RB1i) && ($FL2i != $ +RB2i) && ($FL2i != $WR1i) && ($FL2i != $WR2i) && ($FL2i != $TE1i) && +($FL1i != $FL2i)) #FL1 != If Start + { + ########################################### +#################### + $newMaxSalary = $QB1s + $QB2s + $RB1s + + $RB2s + $WR1s + $WR2s + $TE1s + FL1s; + $newMaxPoint = $QB1p + $QB2p + $RB1p + +$RB2p + $WR1p + $WR2p + $TE1p + $FL1p; + if($newMaxSalary < $maxSala +ry) + { + #print "Current Salary += " . $newMaxSalary . "\n"; + if (!$maxPoint) + { + $maxPoi +nt = $newMaxPoint; + } + else + { + if($max +Point < $newMaxPoint) + { + + $maxPoint = $newMaxPoint; + + print "New Max Points Reached! " . $maxPoint . "\n"; + + print "Here is your team! \n"; + + print "QB1 = " . $QB1i . "\n"; + + print "QB2 = " . $QB2i . "\n"; + + print "RB1 = " . $RB1i . "\n"; + + print "RB2 = " . $RB2i . "\n"; + + print "WR1 = " . $WR1i . "\n"; + + print "WR2 = " . $WR2i . "\n"; + + print "TE1 = " . $TE1i . "\n"; + + print "FL1 = " . $FL1i . "\n"; + + print "FL2 = " . $FL2i . "\n"; + + print "DST = " . $DSTi . "\n"; + + print "******************************************* \n \n"; + + sleep($delay); + }#i +f + }#else + }#if + }#if + ########################################### +######################### + }#FL2 != If End + }#FL2 End + }#FL1 != If End + }#FL1 End + }#DST End }#TE1 +End }#WR2 End }#WR1 End }#RB2 If End }#RB2 End }#RB1 End } }#QB2 End }#QB1 End print "--------------- Projected Team Points = $maxPoints ------------ +--\n"; print "$$$$$$$$$$$$$$$ Projected Team Salary = $salary $$$$$$$$$$$$$$$ +\n\n"; $programStopTime = localtime(); print "Program Start Time - $programStartTime, Program Stop Time - $pr +ogramStopTime \n";
      I love it when a program comes together - jdhannibal
        Wild guess here; but could your sleep($delay); function cause any buffering?

        Like I said, a wild guess. Glancing at it here. That was all that jumped out at me

        HTH

        --Chris

        #!/usr/bin/perl -Tw
        use Perl::Always or die;
        my $perl_version = (5.12.5);
        print $perl_version;
        Are you perchance coding on a widescreen monitor? It might affect your code layout.
        $newMaxSalary = $QB1s + $QB2s + $RB1s + $RB2s + $WR1s + $WR2s + $TE1s ++ FL1s;
        This shouldn't affect anything memory-related, but FL1s is missing a dollar sign there.
        My OCD kicked in and while I could not find any particular memory-related errors, I 'fixed' the program's indentation so that it should be more legible (so the other monks have an easier time looking at it).

        A couple of more obvious errors were fixed, too. Please check that the code still behaves the same.

        You really ought to use warnings at the very least; it might provide pertinent clues to the issue.

        You have "foreach" loops 10 deep in that program. If each one runs only 3 times, that's 60,000 iterations of the inner loop. I would seriously consider rethinking what you're trying to do, or how you're doing it, because an O(n^10) algorithm is usually a recipe for disaster.
        Downvoted!

        Your code is unreadable and you've been begged to reduce indentation, but you didn't react.

Re: Perl Program - Out of memory?
by Laurent_R (Parson) on Nov 15, 2013 at 19:50 UTC
    Please post your code. You may be storing something into memory (explicitly or implicitly), but we can't know without seeing what you do.
Re: Perl Program - Out of memory?
by BrowserUk (Pope) on Nov 15, 2013 at 20:39 UTC

    Please post the output from perl -V


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Perl Program - Out of memory?
by sundialsvc4 (Abbot) on Nov 15, 2013 at 23:47 UTC

    A few things superficially come to mind:

    • You have several identical cases of sort keys $someHash within nested loops.   If the list of keys is big, especially very big, that could consume a lot of memory.   Yet I do not superficially see anything within the accompany logic that mandates the sort in order to produce correct results.
    • It occurs to me that much of this logic appears to be similar to what might be obtained ... at least, in part ... by an SQL INNER JOIN query.   You’ve got a lot of nested loops here, none of which seem to be doing much more than to set-up for the next inner nested loop.   And, well, “if it’s combinations that you want, even with ORDER BY, then SQL is already fantastic at doing that sort of thing.   How much of this “nested loop” logic might be replaceable by a single SQL query, involving many JOINs, which would present all of those combinations to you naturally as part of its result?

    Pursuing the last thought, I frankly do suspect that a lot of this “nested loop” logic could, indeed be expressed as a query which might, indeed, produce many thousands of rows as a “cross-product” between several smaller constituent tables.   (“And so what... that’s what SQL servers do for a living...”)   But this might then serve to rather-drastically reduce the complexity and memory-footprint of your code, which now only has to consume a record-set that is presented to it.

    Note also that “SQL” doesn’t have to imply “a server.”   The SQLite database system, for instance, is built on single-files, and it runs quite nicely on everything from mainframes to cell-phones.   My essential notion here is that maybe you can shove “all that data” out of (virtual...) memory, and into file(s).

      Yes, it is also not clear to me whether sorting the hash keys is necessary, but if it is necessary, you might consider sorting the keys before entering the nested loops and storing them into an array (and walking through the arrays containing the sorted keys rather than the hash keys). I do not know whether it will really reduce memory usage sufficiently, but it will certainly reduce considerably the run time. The hash keys used in the most inner loops might be sorted millions or possibly even billions of times, this is a huge waste of CPU power, as your program is likely to spend most of its running time sorting again and again the same data. Now, to restate, this will certainly also save some memory, but I have no idea whether this will be enough to solve your memory problem.

      One additional point: how many elements do you have in each of your hashes?

Re: Perl Program - Out of memory?
by Laurent_R (Parson) on Nov 16, 2013 at 13:37 UTC

    Hi,

    Further to my idea of presorting the hash keys described in Re^2: Perl Program - Out of memory? (rather that sorting them again and again), I would suggest that you try the following modified program.

    I made the following changes: presorting the hash keys into arrays and using the arrays of keys to walk though the data, added my statements to declare your variables so that you can enable warnings and strictures (I might have missed some, I can't really test or compile).

    Three additional points: (1) I do not see the point of using a delay in your most inner loop, I think you can remove it. (2) The fact that you did not give lexical scope to your $key loop variable is probably a major bug in your program, because I think that the $key variable at one level of nesting gets overwritten by the last value taken by $key within the next nested loop (not entirely sure though, I have not tried in ages to use such loop variable without lexically scoping it, maybe Perl can manage this correctly, but I would be surprised). OTOH, it may have no impact because starting the inner loop is the last thing you do each time, but this is poor and dangerous design. Anyway, this should no be the case with my version above, where each $key variable is lexically scoped within its loop. (But, again, I can't test anything.) (3) Finally, the $salary variable used at the end of the program seems to be coming from nowhere. Using warning and strictures would detect this, just as the fact that you are using $maxPoint and $maxPoints.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1062777]
Approved by BrowserUk
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (14)
As of 2014-12-17 20:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (33 votes), past polls