Beefy Boxes and Bandwidth Generously Provided by pair Networks Cowboy Neal with Hat
Welcome to the Monastery
 
PerlMonks  

Dei ex machina - State of the External Dependency Arts

by Intrepid (Deacon)
on Mar 23, 2013 at 10:08 UTC ( #1025019=perlmeditation: print w/ replies, xml ) Need Help??

Three past Perlmonks discussions in particular sparked this writeup.

  1. 1020332 Updating Config.pm late Feb 2013.
  2. 1020022 Installing Glib on Debian - same author as above, late Feb 2013.
  3. 1023945 this top post, and specifically that offshoot comment.

The last thread above describes scenarios that involve difficulties encountered in creating a correct installation of a CPAN module because of unfulfilled external dependencies (where “external” in this context means provisioned from outside the CPAN installer universe). The first two threads are perlquestions involving a specific example of this kind of struggle. Read on to see where we find ourselves once we enter this wilderness.

Because the CPAN client programs run the module installers, and there are basically 3 module installers being used in recent times, discussion of the sort I am kicking off here is inevitably discussion of those installers.

The 3 installers are Module::Build, ExtUtils::MakeMaker, and Module::Install. The last uses ExtUtils::MakeMaker underneath, but is a significantly different approach from both the perspective of end user experience and from perspective of module author. If this were not the case, it would not be logical to add it to the list containing the first two.

I worked over the last couple of days with exploring the means by which one CPAN module package attempts to address the “external dependency” issue. That's Glib.pm, a perl interface to the cross-platform (Gnome/Gimp/Gtk) C library glib. I found it useful to examine how this package configures itself.

Before going on into detail, I need to mention the Alien:: module namespace on CPAN. AFAICR neither thread cited above had any comments that mentioned Alien:: and omitting a mention of this approach would leave any discussion such as this very incomplete — although I am not going to promote this meditation as a comprehensive survey along the lines of this (Neil Bowers' article is exemplary in its usefulness and interest, imho).

What's always been there

…is one piece of ExtUtils::MakeMaker, EU::Liblist. For newcomers: “EU-MM” (EU::MM), ExtUtils::MakeMaker, is the original Perl5 comprehensive package builder-installer. It has enjoyed a very long useful lifetime. Some have publically prophesied its retirement from service; some would say, several years later, that those forecasts were premature. EU-MM is under the care of the Perl ToolChain Gang now, headed up by the cranky and redoubtable Michael Schwern (yo! \o). I do not think another EU-MM vs. M-B (Module::Build) debate is going to blossom here in this thread. SuperSearch will demonstrate that its already been covered.

I have to mention that I am very fond of EU-MM. And my very least favorite part of EU-MM, the piece I'd like to see completely replaced, is EU-Liblist. I'll mention a few ways in which I think it is kinda horrible, further down. I repeat, for clarity: I want my EU-MM. You'll take it from me when you pry it from my cold dead fingers. But I want an in-place surgery in which the malfunctioning organ within it, EU-Liblist, gets removed deftly and carefully, and perhaps given a nice but short burial service, and then we can all attend a big party celebrating Perl5's entry into the 21st C.

And then there is

ExtUtils::PkgConfig. By the same Brian Manning (XAOC) (CPAN author) who brought us Camelbox Perl and a lot of CPAN Gnome/Gtk goodies. This utility is always described by its author in the same way ("simplisticsuper-simplistic"). I've got it installed to CygwinPerl and to Camelbox and to Strawberry (on my little dev machine, the underpowered Samsung Netbook, running Windows 7).

The question was:

…“Perhaps this is because there is no one single known-good way to check for it on all systems?” (in When cpan returns the dreaded "won't install without force"). Well, this is the Perl culture, and we do, collectively, seem to like (or at least to generate, maybe in spite of ourselves), many ways to do it (whatever “it” is). So it's fair to say that there is no single way that has been devised to cope with configury for “external dependency” software that may or may not be installed. As a moderately knowledgable person in this area, I feel obligated to mention that this does not mean there haven't been any significant attempts to create such a solution. Our EU-PkgConfig seems like an effort that has languished in obscurity. It got mentioned in "Building extensions that access libfoo's foo-config file". The CPAN reverse dependency chart for EU-PkgConfig shows a rather short list of modules that seem to require it. It does not seem to come up very often.

One might wonder why-when so many wild gestures of distress have been witnessed, over the years, as Perl users and authors grapple with the “external dependency” problem. Is it because this is an interface to a tool that is not, itself, written in Perl (AFAIK, pkg-config is written in plain C)? It seems to me that there is sometimes that kind of bias built in to what our Perl culture has become.

The same thread cited above had a comment mentioning a pure-Perl implementation of pkg-config, on CPAN as just PkgConfig. It provides a command-line script that alledgedly emulates the C pkg-config (I saw many discrepancies). It does not replace EU-PkgConfig. I have not even made it work with EU-PkgConfig (at this writing, have not tried). The larger point is this: pkgconfig (the overall utility / approach is spelled pkgconfig and at the command prompt we type pkg-config [args ...]) is a generalized solution to the problem of storing metadata about installed s'w so that automata on the user's system can retrieve it later. That's what this entire "problem" boils down to: it is about the unavoidable requirement that there be metadata about installed s'w available-metadata from outside the CPAN universe (machina) about s'w from outside the CPAN universe-and about accessing that metadata and applying it appropriately when CPAN pkgs are being configured.

What would gtk2-perl do?

Our Glib.pm takes an approach that now seems problematic but instructive to look at at. It uses both EU-PkgConfig and EU-Liblist. That's right. Yep. The package's (currently: Glib-1.280) Makefile.PL is what we'll need to examine. Take a moment to open it, now.

# line 57 my %glibcfg; unless (eval { %glibcfg = ExtUtils::PkgConfig->find ("gobject-2.0 >= $ +build_reqs{glib}"); 1; }) { warn $@; exit 0; }

She looks for a dependency (gobject-2.0) and gracefully aborts if it is not found. She uses the other novel utility package, ExtUtils::Depends, to create an object that will be used to generate arguments to WriteMakefile:

# line 175 our $glib = ExtUtils::Depends->new ('Glib');

We have to code-dive into Depends.pm in order to figure out what is going on there. When we do, we find out that the LIB parameter key which ends up passed onward to EU-MM has been created, and in turn, this means that EU-Liblist is going to come into play. Here's something simple, for a change: the simplest of rules (or explanations, if you prefer):

If LIB is mentioned when WriteMakefile is called, then ExtUtils::Liblist is going to be invoked inside EU-MM.

I implied (If you noticed further above) that it was odd that both EU-Liblist and EU-PkgConfig were being "used" by the Glib.pm configury. It sounds odd because I've introduced EU-Liblist and EU-PkgConfig to the reader as if they do precisely the same thing and one is therefore an alternative to the other. That isn't precisely true, however. What EU-Liblist does is provide a subroutine ext (called as a static class emthod or as an instance object method) which returns a list of strings (and optionally a final value which is an array-ref); these strings are then used by the other parts of EU-MM to generate Makefile directives which are supposed to be optimal, bulletproof, Perl-normalized, platform-aware, benificent, consistent, or some such pile of virtues. The fact is that the argument passed to ExtUtils::Liblist->ext() is already in precisely the format that a gcc-compatible compiler would expect for the linking stage.

This is where I explain how bad EU-Liblist is. This misconceived organ in an otherwise brilliant apparatus violates one of the primary principles of Virtuous Perl (or any other kind of programming): Be liberal in what you accept and strict in what you emit Wikipedia ref.. (Note to readers: do not misinterpret my tone, please. I admire the work put into all of EU-MM. My intent is to point out that this piece of EU-MM more than any other (IMHO, of course) is a throttle point where improvement in the transparency and ease we provide to Perl software authors gets restricted, thwarted. Time and the consistent stream of new authors entering the Perl realm has proven this; perhaps when it was written, it could not have been easily forseen).

A replacement for EU-Liblist would have some of the following virtues:

  • It would be liberal in what it accepts. An author might naively expect to be able to pass a fqpn (fully qualified path name) to such a mechanism, and have it DTRT in its output. Some of the code in EU-Liblist seems to do this, for some OSes. Other kinds of input parameter formulation will occur to readers.
  • It would return an object that could have methods called on it to provide flexible output of the species of parameters that C/C++ compilation and linking stages require (-L - library file directory search specs; -l - library file abbreviated base-name specs; -I - preprocessing / compilation header includefile directory search specs; etc.). What EU-Liblist ext() returns right now is an inflexible and hard to remember list of strings. It does nothing at all in finding library files, for example, that many CPAN authors haven't already painfully written ad hoc code to achieve in their Makefile.PLs. And which EU-PkgConfig facilitates via the surer, better route of metadata discovery.
  • It would self-facilitate the spread of understanding of its own API, conventions, personality, by being made a bit broader tool with a bit more intelligence built in to it, than EU-Liblist has. EU-Liblist provides ext(), mostly for other parts of EU-MM to invoke secretly (behind the curtain). And that's all. As a consequence this obscure piece of EU-MM remains obscure. That's its immutable destiny. A broader tool with a more friendly API and a more liberal approach to what it accepts would self-advertise. Putting on a sexy skirt isn't always a bad thing. At the same time, this is Tool Chain Infrastructure we are talking about here. Code bloat, unreasonable non-Core dependencies, feeping-creaturism and other such malodorous attributes must be deflected.

I'm sure the reader is convinced that something is not pretty or affable about EU-Liblist, although I am equally sure that I've lost a lot of readers at this point. Here's a chance, therefore, to encounter something familiar. Ever seen messages like this spewing across your terminal console? ;-)

'-lgobject-2.0' not found as 'C:/Camelbox/lib\gobject-2.0.a' '-lgobject-2.0' not found as 'C:/camelbox/lib\gobject-2.0.a' '-lgobject-2.0' not found as 'C:\camelbox\lib/CORE\gobject-2.0.a' Note (probably harmless): No library found for -lgobject-2.0 '-lgobject-2.0' not found as 'C:/Camelbox/lib\libgobject-2.0.a' '-lgobject-2.0' not found as 'C:/camelbox/lib\libgobject-2.0.a' '-lgobject-2.0' not found as 'C:\camelbox\lib/CORE\libgobject-2.0.a' Note (probably harmless): No library found for -lgobject-2.0 '-lgobject-2.0' not found as 'C:/Camelbox/lib\gobject-2.0.dll.a' '-lgobject-2.0' not found as 'C:/camelbox/lib\gobject-2.0.dll.a' '-lgobject-2.0' not found as 'C:\camelbox\lib/CORE\gobject-2.0.dll.a' Note (probably harmless): No library found for -lgobject-2.0 '-lgobject-2.0' found as 'C:/Camelbox/lib\libgobject-2.0.dll.a' '-lglib-2.0' not found as 'C:/Camelbox/lib\glib-2.0.a' '-lglib-2.0' not found as 'C:/camelbox/lib\glib-2.0.a' '-lglib-2.0' not found as 'C:\camelbox\lib/CORE\glib-2.0.a' Note (probably harmless): No library found for -lglib-2.0 '-lglib-2.0' not found as 'C:/Camelbox/lib\libglib-2.0.a' '-lglib-2.0' not found as 'C:/camelbox/lib\libglib-2.0.a' '-lglib-2.0' not found as 'C:\camelbox\lib/CORE\libglib-2.0.a' Note (probably harmless): No library found for -lglib-2.0 '-lglib-2.0' not found as 'C:/Camelbox/lib\glib-2.0.dll.a' '-lglib-2.0' not found as 'C:/camelbox/lib\glib-2.0.dll.a' '-lglib-2.0' not found as 'C:\camelbox\lib/CORE\glib-2.0.dll.a' Note (probably harmless): No library found for -lglib-2.0 '-lglib-2.0' found as 'C:/Camelbox/lib\libglib-2.0.dll.a' '-lintl' not found as 'C:/Camelbox/lib\intl.a' '-lintl' not found as 'C:/camelbox/lib\intl.a' '-lintl' not found as 'C:\camelbox\lib/CORE\intl.a' Note (probably harmless): No library found for -lintl '-lintl' not found as 'C:/Camelbox/lib\libintl.a' '-lintl' not found as 'C:/camelbox/lib\libintl.a' '-lintl' not found as 'C:\camelbox\lib/CORE\libintl.a' Note (probably harmless): No library found for -lintl '-lintl' not found as 'C:/Camelbox/lib\intl.dll.a' '-lintl' not found as 'C:/camelbox/lib\intl.dll.a' '-lintl' not found as 'C:\camelbox\lib/CORE\intl.dll.a' Note (probably harmless): No library found for -lintl '-lintl' found as 'C:/Camelbox/lib\libintl.dll.a' '-lgthread-2.0' not found as 'C:/Camelbox/lib\gthread-2.0.a' '-lgthread-2.0' not found as 'C:/camelbox/lib\gthread-2.0.a' '-lgthread-2.0' not found as 'C:\camelbox\lib/CORE\gthread-2.0.a' Note (probably harmless): No library found for -lgthread-2.0 '-lgthread-2.0' not found as 'C:/Camelbox/lib\libgthread-2.0.a' '-lgthread-2.0' not found as 'C:/camelbox/lib\libgthread-2.0.a' '-lgthread-2.0' not found as 'C:\camelbox\lib/CORE\libgthread-2.0.a' Note (probably harmless): No library found for -lgthread-2.0 '-lgthread-2.0' not found as 'C:/Camelbox/lib\gthread-2.0.dll.a' '-lgthread-2.0' not found as 'C:/camelbox/lib\gthread-2.0.dll.a' '-lgthread-2.0' not found as 'C:\camelbox\lib/CORE\gthread-2.0.dll.a' Note (probably harmless): No library found for -lgthread-2.0 '-lgthread-2.0' found as 'C:/Camelbox/lib\libgthread-2.0.dll.a' [output truncated]

That's EU-Liblist. Whether it succeeds in doing its task or not, you'll see such messages if that rigid, joyless, uptight little nun ext() is called. She won't even come to the party without letting loose such an, um, bracing deluge of complaints. And she wonders why nobody really wants to dance with her.

To make a slightly better appliance out of this miserable, archaic organ of EU-MM, one can write Makefile.PL code such as this:

# Adapted from Makefile.PL, Copyright (C) 2003-2009, 2012 by the gtk2- +perl team ... # [...] my %mm_kv = $glib->get_makefile_vars; $::got_lib = $mm_kv{'LIBS'}; # shut EU-Liblist up the only way possible: do not invite her to the # WriteMakefile() party at all: delete $mm_kv{'LIBS'}; # but override `const_loadlibs´ below; otherwise we will be missing M +akefile # declarations that must be provided for package to build. WriteMakefile( NAME => 'Glib', VERSION_FROM => 'lib/Glib.pm', # finds $VERSION ABSTRACT_FROM => 'lib/Glib.pm', # retrieve abstract from module PREREQ_PM => \%PREREQ_PM, XSPROTOARG => '-noprototypes', MAN3PODS => $glib ? \%pod_files : {}, FUNCLIST => \@exports, DL_FUNCS => { Glib => [] }, META_MERGE => \%meta_merge, $glib ? %mm_kv : (), ); # [...] sub MY::const_loadlibs { my ($self) = @_; my $crunch = my $inexpr = $::got_lib; die unless $crunch; my @is = split(q[ ]=> $crunch); my $lgood; my $rectfy = sub { my @ele; for my $lobster_shoes (@_) { $lobster_shoes=~tr{\\}{/}; push @ele, $lobster_shoes; } return @ele }; # Because shoes for lobsters is just wrong, and so are backslashes f +or perl # pathnames ...nearly never needed / right / non-ugly. my $prettier = sub { my $aw = eval 'require Text::Wrap; 1'; if ($@ or !$aw) { return @_ } my @input = map {$rectfy->($_)} @_; my $evcode = <<' INANDOUT'; local $Text::Wrap::columns = 72; local $Text::Wrap::separator = qq[ \\\n]; local $Text::Wrap::huge = 'overflow'; Text::Wrap::wrap( q[ ] x 1, q[ ] x 22, @input); INANDOUT # print STDERR qq[Generated string\n:] . eval $evcode; return eval $evcode; }; $crunch = $prettier->($crunch); my $OSfam = $^O; my $captbuf; my($E,$B,$L,$R,$libpath); { no warnings 'once'; open SAVEERR, ">&STDERR"; close STDERR; $captbuf = ''; open STDERR, '>', \$captbuf or die "Cannot capture STDERR: $!"; if ($^O =~ /mswin/i) { $inexpr = q[:NODEFAULT ].$inexpr; } ($E,$B,$L,$R,$libpath) = $self->ext($inexpr, "0", "1"); close STDERR; open STDERR, ">&SAVEERR"; } $lgood = $prettier->(@$libpath); my $Es = ($E ne $L) ? $E : $lgood; my $Ls = ($self->{CC} =~/gcc/i) ? $crunch : $lgood; my $Bs = ($B =~/\A\s*\z/ ) ? "":$B; if ($E ne $L) { $E = $rectfy->($E); $L = $rectfy->($L); } else { $E = q[]; $L = q[]; } my $mftext = qq{ # Glib.pm on $OSfam depends on these external libraries: # EXTRALIBS =$Es #--# [$E] LDLOADLIBS =$Ls #--# [$L] BSLOADLIBS =$Bs # # The ExtUtils::Liblist program was used to generate these lists from # the input # # $crunch # }; }

Code above only tested on MS Whynn (7)

What was that?

If you replace an organ, you have to manage the re-connection of the replacment organ to the surrounding organs and tissues. What's shown here is not replacement but rather but an intermediate step (or a band-aid) in which we invoke that miserable ext() method only on our own tightly-controlled terms. A padded room, as it were.

The primary duct connecting EU-Liblist to the automated workings of EU-MM is done when the const_loadlibs routine is called (it is an overidable, Make-text -generating subroutine hidden in the bowels …you get the idea). So we provide an override and put the text we massaged out of her insipid churliness to the master orchestrator that is assembling our Makefile. The output is a labelled section in Makefile that looks like this, on one Perl I've got running:

# --- MakeMaker const_loadlibs section: # Glib.pm on MSWin32 depends on these external libraries: # EXTRALIBS = C:/Camelbox/lib/libgobject-2.0.dll.a C:/Camelbox/lib/lib +glib-2.0.dll.a \ C:/Camelbox/lib/libintl.dll.a \ C:/Camelbox/lib/libgthread-2.0.dll.a #--# [] LDLOADLIBS = -LC:/Camelbox/lib -lgobject-2.0 -lglib-2.0 -lintl -lgthr +ead-2.0 #--# [] BSLOADLIBS = # # The ExtUtils::Liblist program was used to generate these lists from # the input # # -LC:/Camelbox/lib -lgobject-2.0 -lglib-2.0 -lintl -lgthread-2.0 #

Notice how we are back where we started. The only Make macro of those three above that is actually used in any recipe (to build anything), for my gcc-built Whynn32 Perl, is LDLOADLIBS. And I've set it to the input (created from the running of EU-PkgConfig) that EU-Liblist's ext() got! Thus proving that this module is nearly as irrelevant to building Glib.pm as an appendix is to a modern human organism.

Readers are urged to carry out modifications to Makefile.PLs of their chosing, on OSes of their choosing, and add comments that refute this conclusion.

Where to from here (or, will this node be a catalyst for development)?

So I've got some ideas about how to provide a bionic replacement for EU-Liblist. These ideas do not have a guaranteed audience, in my estimation. The Alien:: "movement" has a significant head start, and I urge any reader to look into what they're doing.

Disclaimer: any descriptions I've made of software projects (Glib.pm, pkgconfig, etc.) are not to be taken as canonical descriptions providing an accurate or complete charaterization of the software to the reader. The reader is expected to exhibit the Medieval Christian Mortal Sin of Curiosity and do their own further research. This article is not a list of URLs to already well-known projects. Its intended audience consists of thoughtful people who enjoy learning about things (e.g., hackers).

Comment on Dei ex machina - State of the External Dependency Arts
Select or Download Code
Self-Followup: Dei ex machina - State of the External Dependency Arts
by Intrepid (Deacon) on Apr 01, 2013 at 05:21 UTC

    I wrote that there seems some possibility that the ExtUtils::PkgConfig had not been widely adopted by Perlers because it itself is not a utility written in C Perl. That's highly speculative in a writeup in which I tried to be generally objective:

    One might wonder why-when so many wild gestures of distress have been witnessed, over the years, as Perl users and authors grapple with the “external dependency” problem. Is it because this is an interface to a tool that is not, itself, written in Perl.

    Anyway, it turns out that, lurking in the other neighborhoods of F/LOSS software, there is a (non-CPAN, at this writing) project that implements another variant of pkg-config in Perl. It's at this Github repository (as of this writing) and I've forked it to https://github.com/somian/pkg-config.

    I do endorse the OpenBSD Perl pkgconfig

    It's the best one I've found out there. But there's no CPAN / Perl installer infrastructure put together for users in my repo yet. So it is recommended that people with enough knowledge to work that little problem, only, attempt to install from that codebase.

    UPDATE in 2013: Apr 01
    corrected mistatement above. Apr 01, 2013 at 17:30 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://1025019]
Approved by ww
Front-paged by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (8)
As of 2014-04-16 23:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (436 votes), past polls