combination of multiple installed Perls and some environment variables cause segfaults

[dlamb]

Hi Monks,

First (registered) post. This is a bit of a long one--I was thinking things through while typing it up, which I hope helps you.

I have a Mac running 10.6.8. I have the system perl /usr/bin/perl (5.10.0), and also a MacPorts perl /opt/local/bin/perl (5.14.2), the latter is the first in my $PATH. Until recently I had used CPAN to install non-MacPorts modules into the MacPorts tree /opt/local/lib/perl5/site_perl/. That was OK, but I was told that it was a bad idea to have things in the MacPorts tree that MacPorts doesn't know about, so I recently deleted everything there and reinstalled in my home dir /Users/derek/local/lib/perl5. I put export PERL5LIB="/Users/derek/local/lib/perl5" into my ~/.profile, added INSTALL_BASE=/Users/derek/local to the config options in cpan (makepl_arg and similarly for mbuildpl_arg) and I thought all was well. (yes, I'm aware of local::lib, but don't have it installed.)

Now I am checking some code in the PDL test harness, and the code does image format conversion by calling some netpbm programs (in this case, 'pnmquant'). Some of these are Perl scripts themselves. They start with #!/usr/bin/perl. And I am getting some failures that I think I understand and some I don't, so I wanted to get some outside advice at this point. For the rest of this post, please remember that I have now undefined the $PERL5LIB environment variable, for clarity. The test image is named img.pnm

When I run from the command line
pnmquant 256 img.pnm
It produces the expected output. But when I set that PERL5LIB variable:
PERL5LIB=/Users/derek/local/lib/perl5 pnmquant 256 img.pnm, I get a Segmentation fault. That is bad, because that line (I think) emulates best the environment I will normally be running under. If I try to run under the debugger, I get
PERL5LIB=/Users/derek/local/lib/perl5 /opt/local/bin/perl -d /opt/local/bin/pnmquant 256 img.pnm success or
PERL5LIB=/Users/derek/local/lib/perl5 /usr/bin/perl -d /opt/local/bin/pnmquant 256 img.pnm failure depending on the perl used. The latter debugger fails immediately with

dyld: Symbol not found: _Perl_xs_apiversion_bootcheck
  Referenced from: /Users/derek/local/lib/perl5/darwin-thread-multi-2l
+evel/auto/Term/ReadKey/ReadKey.bundle
  Expected in: flat namespace

Trace/BPT trap

If I prepend VERSIONER_PERL_PREFER_32_BIT=no to that line as recommended in this post I get the same dyld error. That post suggested that modules compiled for different perl versions different from the running perl (specifically 32- vs 64-bit) were the culprit. But I don't think that's the case here. I think it's that doing nm ReadKey.bundle |grep _Perl_xs_api on ~/local/lib/...../auto/Term/ReadKey/ReadKey.bundle returns that symbol and doing the same on the 5.10.0 .bundle (in /System/Library/.....etc) does not return that symbol.

If I instead prepend PERL_DL_NONLAZY=1 to the line, then I get no dyld error and the correct output. This may explain why the code in question does not fail when run under make test (which as you know sets that NONLAZY variable), but does when run under perl -Mblib t/testscript.t.

Question 1: Shouldn't PERL_DL_NONLAZY be helpfully causing these sorts of errors, not preventing them like it seems to be doing?

Question 2: Should my directory structure in ~/local/lib/perl5 have perl-version-specific directories, or is that something I would have had to create manually, or does that not matter in this case?

Question 3: The segfaulting seems to be caused because the PERL5LIB prepends a perl-5.14.2 directory to @INC, a ReadKey.pm and ReadKey.bundle are found there, but the perl 5.10 called by the script or by me manually sees a different symbol table in that .bundle file than it is expecting. How can I avoid that, what do you recommend for a path forward out of this mess? Let's assume I can't (or won't) change the Netpbm source code to have the perl scripts start with #!/usr/bin/env perl instead.

Thanks!

[tobyink]

XS modules compiled for one version of Perl are likely to segfault other Perls. (Though recently p5p have been ensuring binary compatibility across minor versions - i.e. 5.16.0, 5.16.1, 5.16.2 and 5.16.3 should all be binary compatible.)

The problem with PERL5LIB is that environment variables get passed to child processes, so when your Perl 5.14 spawns Perl 5.10, Perl 5.10 sees a lib dir that was intended for 5.14.

You could try adding this near the top of your script:

BEGIN { delete $ENV{PERL5LIB} };

Or you could avoid setting that variable to begin with and instead use lib or use the Perl -I command-line option.

Personally though, I'd suggest App::perlbrew which makes managing multiple versions of Perl a breeze.

[dlamb]

Thanks for the reply.

I don't think I want to delete $ENV{PERL5LIB} because none of these scripts are solely "mine", so I don't want to mess anybody else up.

My understanding was that use lib or -I were good for individual scripts or one-offs, but not a good solution for permanently setting up a working environment. But I could be wrong.

Since I already have 2 system perls (legacy 5.8.9 and the aforementioned 5.10.0) and 2 MacPorts perls (5.12 and 5.14), and presumably those MP perls are needed by other programs I have installed so I can't get rid of them, I think adding more Perls with perlbrew would just complicate things.

Maybe getting netpbm to change to /usr/bin/env perl would be the best solution.

[tobyink]

Adding

BEGIN { delete $ENV{PERL5LIB} };

... to the top if the script shouldn't mess anything up. It will only have any effect on child processes launched by the script - e.g. things launched using system() or backticks.

[Anonymous Monk]

Question 1: Shouldn't PERL_DL_NONLAZY be helpfully causing these sorts of errors, not preventing them like it seems to be doing?

PERL_DL_NONLAZY has nothing to do with it, you're mixing two perl versions which aren't compatible, it will never work

5.8.x, 5.10.x, 5.12.x, 5.14.x, 5.16.x ... aren't compatible, you can't load .dll/.so modules compiled for one version in another

5.10.0 not compatibile with 5.10.1, 5.10.0 has known bugs

Question 2: Should my directory structure in ~/local/lib/perl5 have perl-version-specific directories, or is that something I would have had to create manually, or does that not matter in this case?

If you're thinking perl5.16.1 or some such, I think you'd have to add that yourself

$ perl Build.PL --install_base=..\snacks
...
$ build install
Building Data-Dump-Streamer
Files found in blib\arch: installing files in blib\lib into architectu
+re dependent library tree
Installing ..\snacks\lib\perl5\MSWin32-x86-multi-thread\auto\Data\Dump
+\Streamer\Streamer.bs
Installing ..\snacks\lib\perl5\MSWin32-x86-multi-thread\auto\Data\Dump
+\Streamer\Streamer.dll
Installing ..\snacks\lib\perl5\MSWin32-x86-multi-thread\DDS.pm
Installing ..\snacks\lib\perl5\MSWin32-x86-multi-thread\Data\Dump\Stre
+amer.pm
Installing ..\snacks\lib\perl5\MSWin32-x86-multi-thread\Data\Dump\Stre
+amer\_\Printers.pm

Question 3: The segfaulting seems to be caused because the PERL5LIB prepends a perl-5.14.2 directory to @INC, a ReadKey.pm and ReadKey.bundle are found there, but the perl 5.10 called by the script or by me manually sees a different symbol table in that .bundle file than it is expecting. How can I avoid that, what do you recommend for a path forward out of this mess? Let's assume I can't (or won't) change the Netpbm source code to have the perl scripts start with #!/usr/bin/env perl instead.

Install Netpbm into your install_base, add the ..\snacks\bin in your install_base first in path, netpbm will have correct path, problem will be over

Or make the 5.14 perl first in your path

One perl in the path at a time, any PERL* env vars should match first perl in path, no mixing allowed

Some links on debugging these types of problems

Re^3: DBD::mysql fail install check (DYLD_LIBRARY_PATH ldd otool install_name_tool VERSIONER_PERL_PREFER_32_BIT .bundle/.dylib/.so

Re^3: DBD::mysql fail install check (crontab debugging troubleshooting DYLD_LIBRARY_PATH ldd otool ...)

[dlamb]

Re: Question 1: I understand that the different perl versions are incompatible. But then why does doing  PERL5LIB='/Users/derek/local/lib/perl5' pnmquant 256 img.pnm segfault (well, I know why, because that PERL5LIB is for 5.14 and pnmquant uses /usr/bin/perl v5.10) but PERL_DL_NONLAZY=1 PERL5LIB='/Users/derek/local/lib/perl5' pnmquant 256 img.pnm produces the correct result without segfaulting? Seems like PERL_DL_NONLAZY is doing something, but I don't understand what, apparently.

The Netpbm scripts (clarification, just in case: Netpbm is not a Perl module) have /usr/bin/perl hard-coded in their #!, so I don't think changing their install location will affect things.

[trwww]

I don't have any specific answers to your questions, but I do have some relevant suggestions that might help:

I'm not sure why you're setting INSTALL_BASE. Its not necessary when making arbitrary perl installs anyways. Adding the perl you want to use in that session to PATH should be enough.

Its likely that all of these installs you've fiddled with are using the same directory (~/.cpan) to stage and build perl modules. That might be trouble.

You can see how I manage hand built perl installs at Re: 2nd Perl installation on Mac OSX.

[dlamb]

I'm setting INSTALL_BASE because MacPorts does not contain updated versions of every module I need to install, and their recommendation is to put non-MacPorts-installed Perl modules somewhere besides in the MacPorts /opt/local/lib/perl5 tree. And because I have some locally-developed modules that need a home. So anything that comes straight from CPAN goes into INSTALL_BASE. Isn't that the point of that?

Thanks for the link on per-project perl installs. That's an approach I hadn't thought of, I'll give it some thought. But as I replied above regarding Perlbrew, I'm trying to keep the number of perls at a manageable level, and I think I may have already crossed that threshold!

[sundialsvc4]

With both perls in turn, issue the command perl -V (with an uppercase V) and notice the list of library search-paths.   This is the current combined list .. the content of @INC .. as taken from all sources including PERL5LIB.

If you are dealing with two different-generation Perls, as here, these lists ought not to intersect at all.   You need to build Perl yourself to do this, in many cases, since the last few entries in the list are hard-coded at compile time.   That’s my rule-of-thumb opinion only ... the safest and most reliable course, although perhaps the most time-consuming since it studiously starts with nothing in-common with its surroundings.

The issue, of course, is XS ... C-language code that is invoked by a CPAN module.   This stuff goes to the heart of Perl, the so called perlguts, and if it was written for a different Perl it just won’t work.   Modules will detect what version is being used at the time that they are installed, and will install themselves accordingly, but if the @INC lists overlap, a subtle mixture of apples and oranges may still occur.

“Older software” tends to be more problematic since it uses “older modules,” and Perl, like all language systems, does evolve over time.

[dlamb]

Yes, those @INCs do intersect to the extent of my PERL5LIB. Which, obviously, is what is causing the problem with that 5.10 /usr/bin/perl. I'm just not sure how to deal with having a PERL5LIB together with scripts (that I didn't write) that call the system perl. Maybe the answer is "don't".