Solving XS and dyld issues on Perl Mac OS X

[ait]

Hello fellow a monk,

I am considering this as an RFC for a tutorial on Perl Mac OS X so if you think it's useful as a tutorial on PM, make your comment and I will expand it and draft into a general tutorial for Perl Mac OS X.

After some research I discovered that a lot of people have been having problems with Mac OS X and Perl modules that use XS bindings to C even here on PM dyld.

CPAN modules will rarely fail but modules from source or trying to cook your own on OS X may not be so easy. I've been banging my head trying to get an XS module to work for the past day and a half so after getting it working I though it was useful to share how...

The main thing to understand is that Mac OS X will ship with a relatively new version of Perl and some thing depend on this so you shouldn't be using this Perl for your development. This is similar to Solaris in that the base system depends on a very specific version of Perl.

You should then have 2 Perls in your Mac. A good option is MacPorts but there are others, including installing from source. I use MacPorts so

  which perl

Will result in

  /opt/local/bin/perl

Libraries and CPAN modules will install in /opt/local as well keeping it separate from the system Perl.

Thats why it's important to use the newer tendency in shebang lines:

  #!/usr/bin/env perl

Instead of the retrogade:

  #!/usr/bin/perl

In fact in Mac OS X this is very important because it is expected that system Perl is /usr/bin/perl !

OK, so now that we clearly have 2 Perls and our development Perl is NOT the system Perl we try to build or compile an XS module.

I won't go into the details of compiling a dynamic library for Darwin but let me point out for reference that there are some important differences from other Operating Systems. Most notably is the distinction between shared libraries and dynamically loadable modules ("a.k.a. bundles"). On ELF systems both are the same and en in .so, on OS X however, bundles end in .bundle or more confusingly in .so whilst dynamic libraries (the .so we are used to in ELF-based systems) en in .dylib.

The other interesting feature of Perl in Mac OS X is that it supports both 32 and 64 bit code. The exact mechanisms are described here

https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man1/perlmacosx5.10.0.1.html

OK, so now you know that your XS code will probably be using a dylib and the lib will probably be in /opt/local/lib. Again just for reference just in case you want to compile your C library on a Mac here is the difference from "soname" on ELF systems and you should build with something like this in your Makefile:

os := $(shell uname)

libboxr : boxr.c
# MacOSX
ifeq ($(os),Darwin)
    gcc -fno-common -c boxr.c
    gcc -dynamiclib -install_name /opt/local/lib/libboxr.1.dylib \
    -compatibility_version 1.0 -current_version 1.0.0 \
    -o libboxr.1.0.0.dylib boxr.o
#Linux                       
else
    gcc -shared -fPIC -g -Wall -Wl,-soname,libboxr.so.1 \
    -o libboxr.so.1.0.0 -lc boxr.c
endif

Assuming you compiled and installed you .dylib correctly, you should now be able to run hsxs -x foo.h and it should locate foo.h from /opt/local/include

So far so good, but try to make test and you will get an undefined symbol, which is what many people have been getting. This is because the standard Makefile built from Makefile.PL will have PERL_DL_NONLAZY=1 which will force the dynloader to resolve all the DL symbols when Perl fires up, which is of course a good thing for developing and testing your XS code. In fact, the XS distro built by default will already have a couple of tests written for you! Anyway, this undefined symbol can occur for several reasons including:

• The dylib cannot be found
• The dylib cannot be loaded (64/32 bit problems)
• The symbols really don't exist on the dynamic lib

Not surprisingly the solution is actually quite simple. Edit the Makefile.PL and add -lfoo to the LIBS variable before you make the Makefile like so:

WriteMakefile(
    NAME              => 'Boxr',
    VERSION_FROM      => 'lib/Boxr.pm',
[snip]
    LIBS              => ['-lfoo -lbar'],
[snip]

And that's it folks. Hopefully that should fix most of your XS pain on Mac OS X.

Cheers,
--
Alejandro Imass

[kcott]

G'day ait,

Responses to "Are there any major Perl issues with Mac OS X Lion?" may be of use to you.

[sundialsvc4]

Indeed.   The Perl versions that ship with Lion vs. Snow Leopard are entirely different.   You will need to prepare your (earnestly awaited!) tutorial with careful parallel references to both, as both are now in common use and are likely to be for some time to come.

[trwww]

Off topic, but related: I was trying to figure out what percentages of users were on the different OSX versions but my google fu is weak on this one. Seems like Apple would tell us their platform statistics.

I was assuming that in practice there is probably very few people still on Snow Leopard... yet I am still on Snow Leopard so maybe it is more common.

[Anonymous Monk]

I recommend perlbrew over relying on macports' perl. Many other ports rely on it's version of perl, so using a cpan tool with that version can mess things up, just as with the system version. Also, do you have example of which CPAN modules fails on OSX. I've never had any problems with the many XS modules I've installed and developed, except for EV, which was just the author being refusing to workaround a compiler bug.

[ait]

Also, do you have example of which CPAN modules fails on OSX. I've never had any problems...

Only the references to MySQL and Oracle modules and a few other mostly seemed that users were using source instead of the CPAN that came with their distro of Perl. My findings as mentioned in the OP were encountered whilst creating an XS modules on the Mac which for me was a first and I never had these problems in Linux.

I use the CPAN shell all the time with my MacPorts Perl and have never noticed any conflicts with system Perl. Could you please expand a little on how these problems may happen?

[Anonymous Monk]

:) yeah, all those problems are getting the wrong binary packages and trying to mix them with other incompatible binary packages -- nothing perl specific except perl was involved

[Anonymous Monk]

Did that ever get solved? Did you find a way to work around it?

[aufflick]

Also useful when developing/testing is the DYLD_LIBRARY_PATH environment variable.

[trwww]

My contribution for this is at Re: 2nd Perl installation on Mac OSX. I think my comment and your meditation overlap almost completely, but maybe theres something there that could simplify yours.