Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Make Perl use real malloc

by creamygoodness (Curate)
on Mar 15, 2011 at 03:33 UTC ( #893231=perlquestion: print w/ replies, xml ) Need Help??
creamygoodness has asked for the wisdom of the Perl Monks concerning the following question:

Greets,

To catch memory leaks in the Perl bindings for Apache Lucy, we run the entire test suite under Valgrind. It takes a long time to complete, but it's an effective technique for catching problems in our C code.

However, Valgrind doesn't ordinarily catch leaks of Perl/XS data structures. As I understand things, Valgrind overrides malloc/calloc/realloc/free and installs its own versions (which track memory usage). However, if you have your own allocation scheme, as Perl does, Valgrind may not notice when things go awry.

Say that your custom allocator requests blocks in large chunks from the OS, and then frees them during a cleanup phase just prior to process exit. Valgrind will likely notice if you fail to free one of those large chunks -- but it won't notice if memory parceled out by your allocator leaks during ordinary operation. I believe that describes the situation with Perl.

The Test::LeakTrace module from CPAN has caught memory leaks for us that Valgrind has not -- for instance, a qr// structure created from Perl-space that our XS binding code leaked a refcount for. Theoretically, we might consider running our test suite under Test::LeakTrace, just as we run it under Valgrind, in order to perform a thorough check. However, what I would prefer to do is compile a custom debugging Perl without the custom allocator, so that allocation of Perl data structures goes through malloc/free/etc. and our existing usage of Valgrind can catch these leaks.

Looking at the file malloc.c in the Perl source (I'm using the tarball for 5.12.2), it seems that definining the NO_FANCY_MALLOC and PLAIN_MALLOC symbols might achieve this goal.

There are two macros which serve as bulk disablers of advanced features of this malloc: NO_FANCY_MALLOC, PLAIN_MALLOC (undef by default). Look in the list of default values below to understand their exact effect. Defining NO_FANCY_MALLOC returns malloc.c to th +e state of the malloc in Perl 5.004. Additionally defining PLAIN_MALL +OC returns it to the state as of Perl 5.000.

I tried running ./Configure with those options:

./Configure -DDEBUGGING -Dprefix=/usr/local/debugperl \ -DNO_FANCY_MALLOC -DPLAIN_MALLOC -es

It doesn't seem to have worked, so maybe I have to specify those symbols using something other than the command line.

Before I go try again, does our analysis seem sound and our approach seem promising? I haven't yet grokked the details of Perl's deep dark magic memory allocation.

Comment on Make Perl use real malloc
Select or Download Code
Re: Make Perl use real malloc
by ikegami (Pope) on Mar 15, 2011 at 05:58 UTC

    To pass defines the the C compiler rather than the configuration system, use

    sh Configure -Accflags="-D... -D..."

    Note that one of Configure's prompts asks what allocator you want to use. Wouldn't that be a better way to go?

    But I thought valgrind worked with the default configuration on linux?

    Good luck!

      It used to be that the FreeBSD port of Valgrind couldn't work with the system Perl, but that was because the FreeBSD system Perl was built with -Dusemymalloc, which Valgrind can't handle. I don't know of a similar problem with any system Perl in a Linux distro, nor have I seen any problems with a system Perl on OS X. Valgrind works with Perl on all those systems. It also works with Perl on FreeBSD if you custom compile your own Perl without -Dusemymalloc.

      The only Configure prompts I know of relating to malloc() control PERL_MALLOC_WRAP and usemymalloc:

      Do you wish to wrap malloc calls to protect against potential overflow +s? [y] n Do you wish to attempt to use the malloc that comes with perl5? [n]

      If we want to work with Valgrind, we definitely don't want to enable usemymalloc (which is disabled by default).

      The Accflags="-DNO_FANCY_MALLOC -DPLAIN_MALLOC " Configure flag worked as intended -- thanks for the tip! Unfortunately, it did not produce a Perl that allowed us to use Valgrind in place of Test::LeakTrace.

      Maybe what we want is impossible because of how the global destruction phase works in Perl. If we use a debugging Perl and set PERL_DESTRUCT_LEVEL to 2, Perl will clean up all scalars, regardless of their refcounts. If we don't set PERL_DESTRUCT_LEVEL, it drops everything on the floor and lets the process exit() clean things up. There's no middle ground where it leaves only scalars with leaked refcounts behind.

        No matter what allocator Perl uses, valgrind can't detect leaked SVs (e.g. reference cycles, refcount to high) if Perl is always aware of its SVs, and that has to be the case since Perl attempts to free them during global destruction.

        $ valgrind perl -e'my $x; $x = \$x; undef $x if $ARGV[0]' 1 ==21363== Memcheck, a memory error detector ==21363== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et +al. ==21363== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h fo +r copyright info ==21363== Command: perl -emy\ $x;\ $x\ =\ \\$x;\ undef\ $x\ if\ $ARGV[ +0] 1 ==21363== ==21363== Warning: bad signal number 0 in sigaction() ==21363== ==21363== HEAP SUMMARY: ==21363== in use at exit: 80,794 bytes in 609 blocks ==21363== total heap usage: 786 allocs, 177 frees, 93,378 bytes allo +cated ==21363== ==21363== LEAK SUMMARY: ==21363== definitely lost: 0 bytes in 0 blocks ==21363== indirectly lost: 0 bytes in 0 blocks ==21363== possibly lost: 14,538 bytes in 354 blocks ==21363== still reachable: 66,256 bytes in 255 blocks ==21363== suppressed: 0 bytes in 0 blocks ==21363== Rerun with --leak-check=full to see details of leaked memory ==21363== ==21363== For counts of detected and suppressed errors, rerun with: -v ==21363== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 22 from + 7) $ valgrind perl -e'my $x; $x = \$x; undef $x if $ARGV[0]' 0 ==21366== Memcheck, a memory error detector ==21366== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et +al. ==21366== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h fo +r copyright info ==21366== Command: perl -emy\ $x;\ $x\ =\ \\$x;\ undef\ $x\ if\ $ARGV[ +0] 0 ==21366== ==21366== Warning: bad signal number 0 in sigaction() ==21366== ==21366== HEAP SUMMARY: ==21366== in use at exit: 80,794 bytes in 609 blocks ==21366== total heap usage: 786 allocs, 177 frees, 93,378 bytes allo +cated ==21366== ==21366== LEAK SUMMARY: ==21366== definitely lost: 0 bytes in 0 blocks ==21366== indirectly lost: 0 bytes in 0 blocks ==21366== possibly lost: 14,538 bytes in 354 blocks ==21366== still reachable: 66,256 bytes in 255 blocks ==21366== suppressed: 0 bytes in 0 blocks ==21366== Rerun with --leak-check=full to see details of leaked memory ==21366== ==21366== For counts of detected and suppressed errors, rerun with: -v ==21366== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 22 from + 7)
Re: Make Perl use real malloc
by Khen1950fx (Canon) on Mar 15, 2011 at 06:36 UTC
    Whether the analysis seems sound or not, I'd probably not want to return to v5.004. If you configured perl with -DDEBUGGING, and you chose to use the perl malloc, then try this script:
    #!/usr/bin/perl use strict; use warnings; warn('!'); do 'lib/auto/POSIX/autosplit.ix'; warn('!!! "after"');
    Run it like this:
    perl -Dm script.pl
    I ran it with -Dm, but if you want other options, run it with -D for a complete list of options.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://893231]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (8)
As of 2014-08-30 04:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (291 votes), past polls