[Win32] "Free to wrong pool ..." error

[syphilis]

Hi,
This is something that doesn't affect linux, though there could well be other operating systems in addition to Win32 that *are* affected.

Let's say I have a static libfoo library that has a my_foo() function that returns a newly allocated (with malloc) string. In C, we can code as follows:

char * string;
string = my_foo();
// Do stuff
free(string);

No problem with that but, on Win32 perl 5.10 and 5.8, in XS (Inline::C) code that's built against that libfoo library, the same code causes a "Free to wrong pool error..." when the free() call is made.
Remove the free() call, and everything is fine - except, of course, that you've then got memory leaks to contend with.

How can this be dealt with in XS code ? That is, how can this memory be freed in XS code ?

I've written a simple libfoo library that demonstrates the issue (which I can post, if that helps). My simple libfoo library also demonstrates that if the library itself has a function to free the memory, then the problem is solved. That is, if libfoo has a function called foo_free() that looks like this:

void foo_free(char * string) {
     free(string);
}

then, instead of calling free(string) in the XS code, I just call foo_free(string) and everything is fine.

But the real life library I'm looking at (libidn) doesn't appear to have such a function (should it ?). That library has a number of functions that return a newly allocated string. For most of those functions, the way that the memory is supposed to be freed is not specified - but where it is specified, the documentation says that the memory should be freed by calling free(), and I surmise that calling free() is the only way that the memory can be freed. As I've outlined above, that works fine for C programs (on both linux and win32), but not for XS code (on win32 only).

Cheers,
Rob

[Anonymous Monk]

It should have such a function, and I think it does, idn_free

For memory you allocate you should use New/Safefree (see perlclib), but for memory some c-lib allocate you're supposed to use free (or idn_free).

[syphilis]

It should have such a function, and I think it does, idn_free

Bloody hell !!
Is it unreasonable of me to have expected this to be documented in 'libidn.info' (which is distributed with the libidn source) ?

Anyway, Anonymous Monk, you're right (as always :-) ... and I thank you for detecting this.

Cheers,
Rob

[ikegami]

This is something that doesn't affect linux

Did you try on a threaded linux build? This message only applies to threaded builds of Perl if I understand correctly.

instead of calling free(string) in the XS code, I just call foo_free(string) and everything is fine.

Sounds like this bypasses a redefinition of free by Inline::C.

[syphilis]

Did you try on a threaded linux build?

Good point that I hadn't thought of. I was thinking that the difference was one of Linux versus Win32, but I gather you're suggesting it's one of threaded vs unthreaded. My linux builds are unthreaded and my Win32 builds are threaded, so you could be (and most likely are) right.

As it turns out, I *do* have an unthreaded build of perl-5.10.0 on Win32 and, indeed, there's no problem with calling free(string) on that particular build. So it looks like it's an issue on threaded builds of perl *only*.

Sounds like this bypasses a redefinition of free by Inline::C

Not sure what I'm s'posed to deduce from that. The issue I was describing is a general XS issue and not something limited to Inline::C (but you probably knew that). And there's usually no problem with free() in XS/Inline::C if the memory is allocated with malloc(). For instance, there's no problem with the following (in either threaded or unthreaded perl):

use warnings;
use Inline C => <<'EOC';

#include <malloc.h>

void foo() {
     char * string;
     string = malloc(10 * sizeof(char));
     free(string);
}

EOC

foo();
print "All done without error\n";

The problem seems only to arise when it's the external library that allocates the memory.

Cheers,
Rob

[BrowserUk]

The problem seems only to arise when it's the external library that allocates the memory.

That's (probably) because the external library is calling the C-runtime malloc(), but the XS code is calling the XS header redefined free() instead of the C-runtime one.

If you call malloc() from within XS code, that is also redefined to use a Perl header wrapper which means the perl malloc is called and it allocates the memory from it's internal pools (and probably tags it). So when you then call free() (the redefined version), it can check the tags (or cross reference the address or however it determines that it is "freeing from the wrong pool"), and everything matches up, so no message.

But when the memory was allocated in an external library (by the C-runtime), and you then attempt to free it from your XS code, the Perl memory management routines have no record of the address you are trying to free (as it didn't allocate it), so it issues the warning.

One possible answer is to create your own free() (just to confuse matters further:), and place it at the end of your XS/Inline C code. And #undef free before you call it. Something like:

// Your inline C stuff here

...
// where you currently call free() call MyFree()
    MyFree( p );
...


void MyFree( void *p ) {
#undef free

    free( p );
}
//EOF