Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Memory usage of array and hashe variables by multiple inclusion.

by tobias_hofer (Friar)
on Jul 02, 2013 at 10:42 UTC ( #1041995=perlquestion: print w/ replies, xml ) Need Help??
tobias_hofer has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks

I am facing an increasing memory usage over the time (during the application-run) and i am not shure what could be the root-cause.
Ok.. I got here a test-system written in perl (not by me) which is loading test-code (also written in perl)more or less dynamically like this:
$para_return = &{ $tc_module."::".$para_init_func } ();
The test-system itself and the test-code are using the same modules for accessing global variables.
However, some of those modules have some globals which will be accessed/filled during the test run - in example:
our $VERDICT = VERDICT_NONE; our @ACTUAL_ERROR_CODES = (); our %TC_PARAMETER = (); our %TC_PRJ_PARAMETER = (); our @TC_HTML_TEXT = (); our $REPEAT_TC_FLAG = 0; my $ALL_ERR_CODES = {}; my $ErrorHandler = ITF::ErrorHandler->new(); my @SYS_FAIL_ERRORS = qw( 111 112 113 100 160 ); my $current_loglevel; my $initial_loglevel; my $current_TC_ID; my %All_ZipFile_to_Handle; my %All_ZipHandle_to_File; my %Alive_Hosts; my %PUBLIC_VARS; our %EVAL_COLLECTION;
Now the badest. Some of these modules include each other, and modules are loaded several times.

Can modules be loaded several times?

Now my wild guess, in case modules can be loaded several times the global vars stay as they are but during the "use" the local vars and objects are created each time the module is loaded. Thus the memory increases - I guess ?

Am I right or completely wrong?

Is there in perl something like the include-protection in C ?

Comment on Memory usage of array and hashe variables by multiple inclusion.
Select or Download Code
Re: Memory usage of array and hashe variables by multiple inclusion.
by kcott (Abbot) on Jul 02, 2013 at 11:25 UTC

    G'day tobias_hofer,

    "Can modules be loaded several times?"

    require says:

    require demands that a library file be included if it hasn't already been included.

    and use says:

    It is exactly equivalent to

    BEGIN { require Module; Module->import( LIST ); }

    So, assuming you're using use or require to load the modules, the answer's no.

    You may find Test::LeakTrace helpful.

    -- Ken

      Hello Ken,

      Thanks a lot!
      So I can exclude this as root-cause of the memory consumption.
      ;-)
Re: Memory usage of array and hashe variables by multiple inclusion.
by flexvault (Parson) on Jul 02, 2013 at 13:16 UTC

    tobias_hofer,

      I am facing an increasing memory usage over the time (during the application-run) and i am not sure what could be the root-cause.

    When I have this problem, it usually is a global variable that continues to grow. In some cases, I use a subroutine that I call every so many seconds to show the memory status of the variables. Something like this (Note: I did this from memory so it may not compile asis).

    sub ShowGrowth { use Devel::Size qw(total_size); our ( $BigScalar, %BigHash, @BigArray ); my $nokeys = keys ( %BigHash ); my $noitems = $#BigArray; print $DLOG "####### Devel::Size #########\n"; print $DLOG "\$BigScalar: ",total_size(\$BigScalar),"\n"; print $DLOG "\%BigHash: ",total_size(\%BigHash)," No of keys: $no +keys\n"; print $DLOG "\@BigArray: ",total_size(\@BigArray), " No of element +s: $noitems\n"; }
    You may want to add the actual memory location to the log. I have learned the hard way that the variable that I think I'm clearing may be the wrong one. To get the memory address:
    print $DLOG "\$BigScalar: ",total_size(\$BigScalar)," Addr: ",\$Bi +gScalar,"\n";
    Leaking memory problems are never easy to find, but this code may give you some ideas on how to find the problem.

    Regards...Ed

    "Well done is better than well said." - Benjamin Franklin

      Thanks a lot! I am now checking the size of the globals. They seem to increase quite fast due to logging

      Best Regards!
      Tobias
Re: Memory usage of array and hashe variables by multiple inclusion.
by sundialsvc4 (Abbot) on Jul 02, 2013 at 13:51 UTC

    A common cause of memory-leaks in Perl are circular references, whereby references among objects form a “loop” such that the reference-count to the members of the structure never returns to zero and so the memory never gets reclaimed.

    Having “one thing in more than one place” is not a problem ... that’s just an object with multiple references to it ... so long as you arrange things such that, when an object is no longer required, all references to it go away.

    The “virtual memory size” as seen by an operating system does not know anything about the in-use/free status of individual objects as seen by Perl.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1041995]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (7)
As of 2014-12-26 21:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (176 votes), past polls