|No such thing as a small change|
Dei ex machina - State of the External Dependency Artsby Intrepid (Deacon)
|on Mar 23, 2013 at 10:08 UTC||Need Help??|
Three past Perlmonks discussions in particular sparked this writeup.
The last thread above describes scenarios that involve difficulties encountered in creating a correct installation of a CPAN module because of unfulfilled external dependencies (where “external” in this context means provisioned from outside the CPAN installer universe). The first two threads are perlquestions involving a specific example of this kind of struggle. Read on to see where we find ourselves once we enter this wilderness.
Because the CPAN client programs run the module installers, and there are basically 3 module installers being used in recent times, discussion of the sort I am kicking off here is inevitably discussion of those installers.
The 3 installers are Module::Build, ExtUtils::MakeMaker, and Module::Install. The last uses ExtUtils::MakeMaker underneath, but is a significantly different approach from both the perspective of end user experience and from perspective of module author. If this were not the case, it would not be logical to add it to the list containing the first two.
I worked over the last couple of days with exploring the means by which one CPAN module package attempts to address the “external dependency” issue. That's Glib.pm, a perl interface to the cross-platform (Gnome/Gimp/Gtk) C library glib. I found it useful to examine how this package configures itself.
Before going on into detail, I need to mention the Alien:: module namespace on CPAN. AFAICR neither thread cited above had any comments that mentioned Alien:: and omitting a mention of this approach would leave any discussion such as this very incomplete — although I am not going to promote this meditation as a comprehensive survey along the lines of this (Neil Bowers' article is exemplary in its usefulness and interest, imho).
What's always been there
…is one piece of ExtUtils::MakeMaker, EU::Liblist. For newcomers: “EU-MM” (EU::MM), ExtUtils::MakeMaker, is the original Perl5 comprehensive package builder-installer. It has enjoyed a very long useful lifetime. Some have publically prophesied its retirement from service; some would say, several years later, that those forecasts were premature. EU-MM is under the care of the Perl ToolChain Gang now, headed up by the cranky and redoubtable Michael Schwern (yo! \o). I do not think another EU-MM vs. M-B (Module::Build) debate is going to blossom here in this thread. SuperSearch will demonstrate that its already been covered.
I have to mention that I am very fond of EU-MM. And my very least favorite part of EU-MM, the piece I'd like to see completely replaced, is EU-Liblist. I'll mention a few ways in which I think it is kinda horrible, further down. I repeat, for clarity: I want my EU-MM. You'll take it from me when you pry it from my cold dead fingers. But I want an in-place surgery in which the malfunctioning organ within it, EU-Liblist, gets removed deftly and carefully, and perhaps given a nice but short burial service, and then we can all attend a big party celebrating Perl5's entry into the 21st C.
And then there is
…ExtUtils::PkgConfig. By the same Brian Manning (XAOC) (CPAN author) who brought us Camelbox Perl and a lot of CPAN Gnome/Gtk goodies. This utility is always described by its author in the same way ("simplistic …super-simplistic"). I've got it installed to CygwinPerl and to Camelbox and to Strawberry (on my little dev machine, the underpowered Samsung Netbook, running Windows 7).
The question was:
…“Perhaps this is because there is no one single known-good way to check for it on all systems?” (in When cpan returns the dreaded "won't install without force"). Well, this is the Perl culture, and we do, collectively, seem to like (or at least to generate, maybe in spite of ourselves), many ways to do it (whatever “it” is). So it's fair to say that there is no single way that has been devised to cope with configury for “external dependency” software that may or may not be installed. As a moderately knowledgable person in this area, I feel obligated to mention that this does not mean there haven't been any significant attempts to create such a solution. Our EU-PkgConfig seems like an effort that has languished in obscurity. It got mentioned in "Building extensions that access libfoo's foo-config file". The CPAN reverse dependency chart for EU-PkgConfig shows a rather short list of modules that seem to require it. It does not seem to come up very often.
One might wonder why-when so many wild gestures of distress have been witnessed, over the years, as Perl users and authors grapple with the “external dependency” problem. Is it because this is an interface to a tool that is not, itself, written in Perl (AFAIK, pkg-config is written in plain C)? It seems to me that there is sometimes that kind of bias built in to what our Perl culture has become.
The same thread cited above had a comment mentioning a pure-Perl implementation of pkg-config, on CPAN as just PkgConfig. It provides a command-line script that alledgedly emulates the C pkg-config (I saw many discrepancies). It does not replace EU-PkgConfig. I have not even made it work with EU-PkgConfig (at this writing, have not tried). The larger point is this: pkgconfig (the overall utility / approach is spelled pkgconfig and at the command prompt we type pkg-config [args ...]) is a generalized solution to the problem of storing metadata about installed s'w so that automata on the user's system can retrieve it later. That's what this entire "problem" boils down to: it is about the unavoidable requirement that there be metadata about installed s'w available-metadata from outside the CPAN universe (machina) about s'w from outside the CPAN universe-and about accessing that metadata and applying it appropriately when CPAN pkgs are being configured.
What would gtk2-perl do?
Our Glib.pm takes an approach that now seems problematic but instructive to look at at. It uses both EU-PkgConfig and EU-Liblist. That's right. Yep. The package's (currently: Glib-1.280) Makefile.PL is what we'll need to examine. Take a moment to open it, now.
She looks for a dependency (gobject-2.0) and gracefully aborts if it is not found. She uses the other novel utility package, ExtUtils::Depends, to create an object that will be used to generate arguments to WriteMakefile:
We have to code-dive into Depends.pm in order to figure out what is going on there. When we do, we find out that the LIB parameter key which ends up passed onward to EU-MM has been created, and in turn, this means that EU-Liblist is going to come into play. Here's something simple, for a change: the simplest of rules (or explanations, if you prefer):
If LIB is mentioned when WriteMakefile is called, then ExtUtils::Liblist is going to be invoked inside EU-MM.
I implied (If you noticed further above) that it was odd that both EU-Liblist and EU-PkgConfig were being "used" by the Glib.pm configury. It sounds odd because I've introduced EU-Liblist and EU-PkgConfig to the reader as if they do precisely the same thing and one is therefore an alternative to the other. That isn't precisely true, however. What EU-Liblist does is provide a subroutine ext (called as a static class emthod or as an instance object method) which returns a list of strings (and optionally a final value which is an array-ref); these strings are then used by the other parts of EU-MM to generate Makefile directives which are supposed to be optimal, bulletproof, Perl-normalized, platform-aware, benificent, consistent, or some such pile of virtues. The fact is that the argument passed to ExtUtils::Liblist->ext() is already in precisely the format that a gcc-compatible compiler would expect for the linking stage.
This is where I explain how bad EU-Liblist is. This misconceived organ in an otherwise brilliant apparatus violates one of the primary principles of Virtuous Perl (or any other kind of programming): Be liberal in what you accept and strict in what you emit Wikipedia ref.. (Note to readers: do not misinterpret my tone, please. I admire the work put into all of EU-MM. My intent is to point out that this piece of EU-MM more than any other (IMHO, of course) is a throttle point where improvement in the transparency and ease we provide to Perl software authors gets restricted, thwarted. Time and the consistent stream of new authors entering the Perl realm has proven this; perhaps when it was written, it could not have been easily forseen).
A replacement for EU-Liblist would have some of the following virtues:
I'm sure the reader is convinced that something is not pretty or affable about EU-Liblist, although I am equally sure that I've lost a lot of readers at this point. Here's a chance, therefore, to encounter something familiar. Ever seen messages like this spewing across your terminal console? ;-)
That's EU-Liblist. Whether it succeeds in doing its task or not, you'll see such messages if that rigid, joyless, uptight little nun ext() is called. She won't even come to the party without letting loose such an, um, bracing deluge of complaints. And she wonders why nobody really wants to dance with her.
To make a slightly better appliance out of this miserable, archaic organ of EU-MM, one can write Makefile.PL code such as this:
Code above only tested on MS Whynn (7)
What was that?
If you replace an organ, you have to manage the re-connection of the replacment organ to the surrounding organs and tissues. What's shown here is not replacement but rather but an intermediate step (or a band-aid) in which we invoke that miserable ext() method only on our own tightly-controlled terms. A padded room, as it were.
The primary duct connecting EU-Liblist to the automated workings of EU-MM is done when the const_loadlibs routine is called (it is an overidable, Make-text -generating subroutine hidden in the bowels …you get the idea). So we provide an override and put the text we massaged out of her insipid churliness to the master orchestrator that is assembling our Makefile. The output is a labelled section in Makefile that looks like this, on one Perl I've got running:
Notice how we are back where we started. The only Make macro of those three above that is actually used in any recipe (to build anything), for my gcc-built Whynn32 Perl, is LDLOADLIBS. And I've set it to the input (created from the running of EU-PkgConfig) that EU-Liblist's ext() got! Thus proving that this module is nearly as irrelevant to building Glib.pm as an appendix is to a modern human organism.
Readers are urged to carry out modifications to Makefile.PLs of their chosing, on OSes of their choosing, and add comments that refute this conclusion.
Where to from here (or, will this node be a catalyst for development)?
So I've got some ideas about how to provide a bionic replacement for EU-Liblist. These ideas do not have a guaranteed audience, in my estimation. The Alien:: "movement" has a significant head start, and I urge any reader to look into what they're doing.
Disclaimer: any descriptions I've made of software projects (Glib.pm, pkgconfig, etc.) are not to be taken as canonical descriptions providing an accurate or complete charaterization of the software to the reader. The reader is expected to exhibit the Medieval Christian Mortal Sin of Curiosity and do their own further research. This article is not a list of URLs to already well-known projects. Its intended audience consists of thoughtful people who enjoy learning about things (e.g., hackers).