[Solved] Perl with Swig C module not releasing memory

[megaframe]

I have a some C code that's malloc'ing a bunch of memory. Using swig I created some simple perl bindings and call out this function in perl. I also have a memory free block in the C code.

When I compile and run just the C code with valgrind, their's no issues. It all gets created and free'd. When I use it in perl and call the destructor it never actually releases the memory.

To test this I wrote up a simple test case of what I was doing.

C code, lib.h

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <errno.h>
typedef struct {
  int index;
  double value;
} x_space;

C code, lib.c

include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
#include <errno.h>
#include "lib.h"
#define Malloc(type,n) (type *)malloc((n)*sizeof(type))
x_space* new_space(){
  x_space *space = Malloc(x_space,1000);
  return space;
}
void destroy_space(x_space *space){
  free(space);
}

swig binding, lib.i

/* lib.i */
%module libspace
%{
#include "lib.h"
extern x_space* new_space();
extern void destroy_space(x_space* space);
%}
extern x_space* new_space();
extern void destroy_space(x_space* space);

Perl code

#!/usr/bin/perl
use libspace;
my $space = libspace::new_space();
libspace::destroy_space($space);
exit;

Compiling/Running

swig -perl lib.i
g++ -Wall -Wconversion -fPIC -c -I/usr/lib/perl/5.14.2/CORE lib.c lib_
+wrap.c
g++ -shared lib.o lib_wrap.o -o libspace.so
perl -I`pwd` test.pl

Is their something else I should be doing to get the .so to release this memory?

UPDATE: So it seems the free is working but its releasing the memory back to perl and not to the system. I'm consuming some 40G of memory in the tool I was using in threads. Safest solution was to fork() and exit when operation was complete. Only other thing would have been to reuse the memory in another thread. If I'm wrong and it should go back to system let me know.

UPDATE 2: Found my issue. In the original code I'm wrapping around it uses a single malloc for this giant data series (in the multiple of gigabytes). I can't do that as I don't know the size of the data till I'm done loading it. So I used a linked list to dynamically allocate the space as the data is fed in from perl. These mallocs are small so glibc isn't releasing them back to the system even when several gigabytes are free'd. To fix the issue I added "#include <malloc.h>" to the header and "malloc_trim(0)" to the destroy call.

[bulk88]

You didn't post the outputted C code of SWIG. A WAG says to add "#define NO_XSLOCKS" before the first perl header is included. Perl redefines by preproc malloc and free to its own safesysmalloc and safesysfree.

[megaframe]

Sorry. Figured it could be generated from that. I'd post it but it's some 1938 lines of code.

...but didn't think to look at that I'll poke around see if I can find out if perl is overwriting the malloc/free functions. Also what's 'WAG'?

[bulk88]

http://en.wikipedia.org/wiki/Wag_%28disambiguation%29

WAG, an acronym for "wild ass guess", an estimate based on experience or similarity

Use readmore tags to post long code, dont force the moderators to do it for you. To find out if perl redefined your malloc/free functions, make a .i file from the SWIG outputted .c file. It will be half a MB so don't post the whole thing here, just an function or 2. Your Malloc macro will be expanded to the actual C func symbol that will be called, which might be a perl allocator, not your C compiler's malloc.

[Khen1950fx]

I experimented with with lib.i. This is untested, but it's what I have so far:

/* lib.i */ % module libspace %{

    #include "lib.h"
    extern x_space * new_space();
        extern void destroy_space( x_space * space );

        #ifdef PERL_OBJECT
        #define MAGIC_CLASS _wrap_libspace_var::
        class _wrap_libspace_var : public CPerlObj {
    public:

        #else
        #define MAGIC_CLASS
        #endif
        SWIGCLASS_STATIC
            int swig_magic_readonly( pTHX_ SV * SWIGUNUSEDPARM(sv),
            MAGIC * SWIGUNUSEDPARM(mg) ) {
            MAGIC_PPERL croak("Value is read-only.");
                return 0;
            }

            #ifdef PERL_OBJECT
        };

        #endif

        #ifdef __cplusplus
        extern "C" {

        #endif
        XS(_wrap_new_space) {
            {   x_space * result = 0;
                    int argvi = 0;
                    dXSARGS;

                    if ( ( items < 0 ) || ( items > 0 ) ) {
                    SWIG_croak("Usage: new_space();");
                }
                result = ( x_space * ) new_space();
                    ST(argvi)
                    = SWIG_NewPointerObj( SWIG_as_voidptr(result),
                    SWIGTYPE_p_x_space, 0 | 0 );
                    argvi++;
                    XSRETURN(argvi);
                fail:
                    SWIG_croak_null();
            }
            }

            XS(_wrap_destroy_space) {
            {   x_space * arg1   = ( x_space * ) 0;
                    void * argp1 = 0;
                    int res1  = 0;
                    int argvi = 0;
                    dXSARGS;

                    if ( ( items < 1 ) || ( items > 1 ) ) {
                    SWIG_croak("Usage: destroy_space(space);");
                }
                res1 = SWIG_ConvertPtr( ST(0), &argp1, SWIGTYPE_p_x_sp
+ace,
                    0 | 0 );
                    if ( !SWIG_IsOK(res1) ) {
                    SWIG_exception_fail( SWIG_ArgError(res1),
                        "in method '" "destroy_space" "', argument " "
+1"
                            " of type '" "x_space *""'" );
                }
                arg1 = ( x_space * ) (argp1);
                    destroy_space(arg1);

                    XSRETURN(argvi);
                fail:

                    SWIG_croak_null();
            }
            } %}

Does that help?

[megaframe]

I'm not entirely sure, I'm new to swig. What's all the extra definitions for?

I can blind try them but I'd like to understand them so I know how I'm implementing things, since this is all part of much larger code.

[Anonymous Monk]

public service announcement,

please don't try to analyze this statement by Khen1950fx, Khen1950fx, he just copy/pasted stuff, he doesn't know what it does, what its supposed to do, he's never used anything like it before, has no understanding of it

[Anonymous Monk]

So it seems the free is working but its releasing the memory back to perl and not to the system .. If I'm wrong and it should go back to system let me know.

:) Its S.O.P., memory is returned to the system at perl's discretion, see Mini-Tutorial: Perl's Memory Management and perlclib. Perl probably overrides/redefines free/malloc (: and then swig does stuff -- I don't swig :) but you should probably stick with New/Safefree as perclib recommends