chuckd has asked for the wisdom of the Perl Monks concerning the following question:

Can anyone tell me what are Perl memory leak symptoms? I'd like to know if I need to install a memory leak module to check? Any symptoms related to the perl58.dll failing, weird actions the compiler would make, memory issues, other unusual script ending issues? I have a Perl script that's just ending before its supposed to and blowing up my system memory usage from 600 to 900+ MB instantly and then just quitting on me. And the funny thing it's failing on a line that holds nothing more than an type int counter, where the counter is only at approximately 1,040,770. Does this sound like a memory leak?
  • Comment on What are the symptoms of a memory leak?

Replies are listed 'Best First'.
Re: What are the symptoms of a memory leak?
by samtregar (Abbot) on Feb 07, 2009 at 07:37 UTC
    A "memory leak" is just a bug that causes a program to lose memory. For example, here's some Perl code that leaks memory:

    while (1) { my $b; my $a = \$b; $b = \$a; }

    If you run that code it will eat all your system memory and then (if you're lucky) die. If you're unlucky your OS might decide to kill something else and cause bad things to happen.

    Without seeing your code, or at least getting a good description of it, there's absolutely no way any of use can tell you if you have a memory leak. That could be what's killing your program, but I'd guess not - 300MB isn't very much memory really.


Re: What are the symptoms of a memory leak?
by gone2015 (Deacon) on Feb 07, 2009 at 13:26 UTC

    I sense your frustration and feel for you.

    I would expect a leak to cause ever increasing memory use, at some rate related to the program's demand for new memory, and caused by a failure either to release memory properly or an inability to reuse memory that has been released.

    I agree that it is hard to see why memory use should suddenly balloon when processing $cnt++. So hard, in fact, that I would assume it's an artifact of the tools you have to observe the effects. I note from other postings that the $cnt++ is both immediately before and immediately after a subroutine exit. On subroutine exit Perl will be doing memory things, which look more plausible suspects. Mind you, the effect may have occurred earlier, but was not immediately visible.

    I also note from other postings that there is a hash involved in this problem. I know that the growth of a hash can cause sudden jumps in memory use. When an empty hash is created it is allocated a small number of chain bases. As the hash grows, that number is doubled as the number of entries approaches the number of bases. You can see how a large growing hash could cause a jump in memory demand.

    Some more background, based on observation:

    As far as I can see, during each growth step, Perl needs the previous chain bases and the new set at the same time: releasing the previous chain bases when it has finished rearranging the contents of the hash across the new bases. So a hash that's grown to 256M of chain bases, has a trail of 128M, 64M, 32M and so on behind it.

    Perl's memory management does not (as far as I know) attempt any kind of memory reorganisation in order to reclaim memory. So, when memory is released it leaves holes, all or part of which may be reused later -- memory will fragment. Much of the time, when lots of items of similar size are repeatedly allocated and released, things remain reasonably stable. You can see how a growing hash will leave holes of increasing size across the memory pool. A hash that grows up to some size, is deallocated and then grows again, may reuse the holes it used last time. But if one of "its" holes has had even a small piece chewed out of it, a new piece of memory will be required for the hash when it reaches its largest size.

    Now, different systems behave differently. Perl does something different when allocating very large lumps of memory. As the hash grows, Perl will allocate memory for it from a "general pool". When it gets big enough, it will go and get a separate lump of memory from the system. Further, Perl does not appear give back those lumps, presumably hoping to reuse it later.

    However, these unused lumps are only occupying virtual memory space, not real memory. System tools will show you the virtual space allocation, and may show you the real memory occupation. A large lump of memory which was used, but is no longer in use, will at first occupy some real memory but that will reduce over time as the system recycles it.

    Given an apparently intractable and mysterious problem, I would set about trying to cut down the program that demonstates the problem: removing as much as possible, until I find something that cannot be removed without the problem going away.

    Unfortunately, this does not instantly solve the problem. Though, from what I have seen so far, I would be eliminating stuff not related to the growth of a hash.

      Example. The next Perl program run differently according to the operating system.

      for ( 1 .. 100_000 ) { $x .= ('x' x 1_000); }
Re: What are the symptoms of a memory leak?
by jethro (Monsignor) on Feb 07, 2009 at 14:29 UTC

    Read your other posts after Oshallas hint. At the moment you step past incrementing the counter the next value is already read from $DATFILE. That is the culprit, not your '$cnt++'.

    I just tested it with the perl debugger (perl -d) to be sure, and as expected, the while-test is not an extra step but just gets done behind the scene. So when you step over $cnt++;, the debugger really does $cnt++; while (<$DATFILE>) {. And if that file has a really long line, that might just be enough to send your script packing.

    You might try this on your data to see if there is an overly long line in it:

    while (<$DATFILE>) { print $cnt,' ',length($_),"\n" if ($length>10_000); $cnt++;
Re: What are the symptoms of a memory leak?
by Anonymous Monk on Feb 07, 2009 at 07:26 UTC
    I have a Perl script that's just ending before its supposed to and blowing up my system memory usage from 600 to 900+ MB instantly and then just quitting on me.
    So your program is doing what you wrote, not what you think you wrote?

    Define memory leak, describe your algorithm, and see Memory leak

Re: What are the symptoms of a memory leak?
by explorer (Chaplain) on Feb 07, 2009 at 18:33 UTC

    At windows, see the process manager. When you see the disk activity high and cpu load very low, then you run with a memory leak.

    Solution: remove Windows and install Linux. The memory leak will be, but more later. Or never.

    (based into personal experience).