On Yaks and the Shaving Thereof - finding exported symbols of a C library

[Corion]

After finding some changes that Sniffer::HTTP needed, I tried to run its test suite. My Strawberry Perl installation was missing some modules, since I hadn't touched Sniffer::HTTP for some time. Especially, it needed Net::Pcap. So I went, and downloaded the WinPCap driver and developer pack from http://www.winpcap.org/devel.htm and installed them. Then I found that I had already reported about four months ago that Net::Pcap 0.16 with libpcap 1.0 does not compile on Windows. So, I decided to look deeper into the problem:

cpanm --look Net::Pcap

...dropped me in the unpacked distribution directory. And fair enough, running the Makefile.PL with the appropriate parameters showed me that it did not find any of the "interesting" functions of the library. In fact, I found that DynaLoader loads DLLs on Windows (as expected) but seems to load .a files and other object files on other operating systems . At least, that's what the usage in the Makefile.PL suggests to me:

sub have_functions {
    my @funcs = ();
    print "detecting available functions... ";

    my @paths = DynaLoader::dl_findfile(qw(-lpcap));
    my $libref = DynaLoader::dl_load_file($paths[0]);

    for my $func (@_) {
        my $symref = DynaLoader::dl_find_symbol($libref, $func);
        push @funcs, $func if defined $symref
    }

    print "ok\n";
    return @funcs
}

...

# Check that '-lwpcap' exports 'pcap_open_live'
print have_functions('pcap_open_live');

So, "all" I had to do was to find a suitable replacement (or suitable munging) of that code to find whether an object file exports a certain symbol or not.

I ended up with the following, fairly ugly code:

sub have_functions {
    my @funcs = ();
    print "detecting available functions... ";

    my @paths = DynaLoader::dl_findfile(split /\s+/, $options{LIBS});
    die "Couldn't find any library file satisfying '$options{LIBS}'"
        unless @paths;
    my $libfile = $paths[0];
    
    # On Win32, we assume that the lib file will not be statically lin
+ked
    # but will be a thin wrapper for a similarly named .dll file.
    # This is not universal but works in many cases
    # This assumes that a library -l$foo will map to lib$foo.a
    # through DynaLoader. We then try to find and load $foo.dll in $EN
+V{PATH}
    if ($has_Win32) {        
        (my $dll = basename $libfile) =~ s/\.\w+$//;
        $dll =~ s/^lib//;
        $dll .= '.dll';
        
        ($libfile) = grep { -f } map { File::Spec->catfile($_,$dll) } 
+File::Spec->path;
        die "'$dll' not found in PATH"
            unless $libfile;
    };
    warn "Using '$libfile' as potential symbol candidate";

    my $libref = DynaLoader::dl_load_file($libfile);
    warn "Couldn't load $libfile via DynaLoader ($^E)"
        unless $libref;
    for my $func (@_) {
        my $symref = DynaLoader::dl_find_symbol($libref, $func);
        push @funcs, $func if defined $symref;
        #print "$func", $symref ? "" : " NOT", " found\n";
    };

    print "ok\n";
    return @funcs
}

That code munges a found .a file and infers the name of the DLL that belongs to it, at least for gcc. Then it searches that DLL in $ENV{PATH} and uses DynaLoader to load and inspect that DLL.

My question is whether anybody knows how to extract the symbols of a library. I'm sure that there must be a much better, Perlish way to find out whether an object file (or a C header) exports a certain symbol or not. I briefly toyed with the idea of parsing the output of nm $libfile, but that will break on MSVC, and also, I know nothing about the different incarnations of nm across operating systems. The only relevant module I found is ExtUtils::FindFunctions by Maddingue and it (unsurprisingly) suffers the same problems, as Maddingue also is the author of Net::Pcap.

PS: The patch is submitted so that should make Net::Pcap compile again on Windows (and elsewhere?), and hopefully, a new release of Net::Pcap will include these.

[Anonymous Monk]

At least, that's what the usage in the Makefile.PL suggests to me

FWIW, DynaLoader does not load .a files on other platforms. Net::Pcap perhaps links against a static library (libpcap.a), but DynaLoader should still load dynamic library auto/Net/Pcap.so

I'm sure that there must be a much better, Perlish way to find out whether an object file (or a C header) exports a certain symbol or not.

Maybe the configure/autotools approach, see ExtUtils::CChecker, Objects in this class provide an extension around ExtUtils::CBuilder to simplify the creation of a .c file, compiling, linking and running it, to test if a certain feature is present.

I briefly toyed with the idea of parsing the output of nm $libfile, but that will break on MSVC, and also, I know nothing about the different incarnations of nm

How about you pack your own copy of nm or objdump? Its dumpbin for msvc.

[vkon]

if you're fine with command line, then

dumpbin /exports perl513.dll

could provide you with needed information with a quite simple text output that should be enough for you

....
         37   24 002126B4 PL_collation_ix
         38   25 00213AC8 PL_collation_name
         39   26 002100E4 PL_collation_standard
         40   27 00213A64 PL_collxfrm_base
         41   28 002100E0 PL_collxfrm_mult
         42   29 00213440 PL_colors
         43   2A 00213544 PL_colorset
....

I've browsed C API and haven't found a function that lists functions from a DLL
I seem to remember that one could get a function by ordinal number (@3) but these must be false memories implanted by someone.

Win32::API is closest that I could advice, and this also suggests you know name of your function.

[BrowserUk]

You can. See GetProcAddress(), and the references to "ordinal". But that only allows you to access the addresses of the exported routines, not their names.

[vkon]

Aha...

As long as GetProcAddress is in the core, you even can use it from Perl...
but - beware: perldoc win32

    Win32::GetProcAddress(INSTANCE, PROCNAME)
        Returns the address of a function inside a loaded library. The
        information about what you can do with this address has been l
+ost in
        the mist of time. Use the Win32::API module instead of this
        deprecated function.

anyway, you could do this in XS, then, but still - I would prefer parsing output of 'dumpbin' command...

update still no way to have function name by ordinal, BTW...
:(

[VinsWorldcom]

UPDATE: I hate answering my own questions on here, but after playing with this all morning with no luck, a posted question and then lunch; I returned with a new direction and found this:

http://sourceforge.net/projects/mingw-w64/forums/forum/723797/topic/3882579

Luckily, I had a copy of gendef.exe from a MinGW-64 (4.4.5) that I have installed as gendef.exe doesn't seem to be in c:\strawberry\c\bin. dlltool is included with Strawberry, so:

NOTE: pexports.exe is included with Strawberry and the output is the same as gendef.exe, so one could just use pexports.exe instead of gendef.exe.

NOTE: There is a wpcap.dll installed in C:\Windows\system32 and C:\Windows\SysWOW64. I used the latter.

VinsWorldcom@C:\Users\VinsWorldcom\tmp> cp c:\Windows\SysWOW64\wpcap.d
+ll .

VinsWorldcom@C:\Users\VinsWorldcom\tmp> \Min\MinGW-64\bin\gendef.exe w
+pcap.dll
 * [wpcap.dll] Found PE image

VinsWorldcom@C:\Users\VinsWorldcom\tmp> which dlltool
C:\strawberry\c\bin\dlltool.exe

VinsWorldcom@C:\Users\VinsWorldcom\tmp> dlltool --as-flags=--64 -m i38
+6:x86-64 -k --output-lib libwpcap.a --input-def wpcap.def

VinsWorldcom@C:\Users\VinsWorldcom\tmp> ls -al libwpcap.a

 Directory of C:\Users\VinsWorldcom\tmp

03/29/2011  02:27 PM            65,656 libwpcap.a
               1 File(s)         65,656 bytes

VinsWorldcom@C:\Users\VinsWorldcom\tmp> ls -al WpdPack\Lib\libwpcap.a

 Directory of C:\Users\VinsWorldcom\tmp\WpdPack\Lib

06/25/2010  10:02 AM            54,276 libwpcap.a
               1 File(s)         54,276 bytes

VinsWorldcom@C:\Users\VinsWorldcom\tmp> ls -al WpdPack\Lib\x64\libwpca
+p.a
File Not Found

 Directory of C:\Users\VinsWorldcom\tmp\WpdPack\Lib\x64


VinsWorldcom@C:\Users\VinsWorldcom\tmp> cp libwpcap.a WpdPack\Lib\x64

I used gendef.exe and dlltool.exe to create a new libwpcap.a static library from the 64-bit wpcap.dll. Why isn't there one included in the WpdPack/Lib/x64 directory already - who knows? In any case, once the new static library was copied into place, I re-ran 'perl Makefile.PL' with the following adjustment:

perl Makefile.PL INC=-IC:/Users/VinsWorldcom/tmp/WpdPack/Include "LIBS
+=-LC:/Users/VinsWorldcom/tmp/WpdPack/Lib/x64 -lwpcap"

The subsequent 'dmake' worked this time, 'dmake test' had a few failures (similar to my WinXP experience). After a 'dmake install' a quick test script works fine:

VinsWorldcom@C:\Users\VinsWorldcom\tmp> more test.pl
use Net::Pcap qw(:functions);

@devs = pcap_findalldevs(\%devinfo, \$err);
for my $dev (@devs) {
    print "$dev : $devinfo{$dev}\n"
}

VinsWorldcom@C:\Users\VinsWorldcom\tmp> perl test.pl
\Device\NPF_{7C974F8C-6E64-44D9-B5F5-BE1242FA897D} : Intel(R) PRO/1000
+ MT Network Connection

VinsWorldcom@C:\Users\VinsWorldcom\tmp>

If this node goes completely unnoticed, it will at least serve as a personal archive for the next time I'm doing this on Win7.

ORIGINAL POST
Read more... (14 kB)

[Corion]

Thanks again for posting these instructions, they helped me build Net::Pcap 0.18 on Windows 7 with Strawberry Perl 5.20 64bit.

I've applied these changes to https://github.com/Corion/Net-Pcap and created a pull request as https://github.com/maddingue/Net-Pcap/pull/7.

Some tests still fail for me with the changes above but I will investigate later.

[sethlwilson]

Thanks so much for posting this! I'd been stuck for days trying to figure out why Net:Pcap wouldn't build for me even with the patch. Your solution worked perfectly!