RFC: Portable Alien Library System

[xdg]

This post was orginally made as Portable Alien Library System on win32.perl.org. It is reposted here for wider commentary. Some additional design details, existing commentary and references can be found on that page.

Synopsis

• Modules using "libfoo" list Alien::Foo as dependency
• Alien::Foo namespace used to manage external libraries
• Alien library files contained in the corresponding auto directory (auto/Alien/Foo)
• Alien modules provide an API for CC flags in Makefile.PL/Build.PL
• Alien modules verify library availability (e.g. in platform or Alien standard directories)
• Alien modules responsible for installing binary library files when necessary
• Alien modules support binary package download from a repository
• Alien modules support source compilation or binary re-packaging to the binary package format used for installation

Introduction

The Vanilla Perl Project provides a Win32 Perl and bundled compiler to build Perl modules from CPAN rather than relying on binary module package systems like PPM. Early feedback on the project demonstrate the many popular Perl modules that have dependencies on external libraries -- ones often assumed to exist on a Unix-based operating system.

This document describes the Portable Alien Library System (PALS) -- a framework for external library dependency specification and resolution. While its inspiration is drawn from Win32, its design is intended to provide benefits to module authors and users on all platforms.

Requirements

This list of functional objectives is adapted from the original Alien module and commentary in External Library Handling at win32.perl.org.

Dependency specification

The solution must provide a way for Perl module authors to specify a dependency on an external library. An important consideration is how a solution integrates with or diverges from existing CPAN methods and tools.

Library detection

Libraries may be pre-installed in well known locations (e.g. "/lib") or may be installed ad hoc. The solution will need to detect if a library (and associated headers) are available and identify the associated paths.

Library configuration

Given the wide variety of locations for libraries, the solution needs to provide an abstraction layer to other tools that ensures proper configuration. Examples include:

• Makefile.PL: compiler flags, including library and header paths
• Shared/dynamic library loading: libraries in non-standard locations may need changes to environment settings such as PATH or LD_LIBRARY_PATH to ensure libraries are found at runtime
• Other metadata: library metadata, such as version, may be required to manage dependencies

Binary library package generation

External library compilation from source can be substantially more time-consuming than typical XS compilation for Perl modules. Providing binaries of external libraries (whenever possible) from an online repository will provide a more user-friendly experience.

Binary library package installation

Whether compiled from source or downloaded as a binary, external modules will need to be installed such that they may be located later for library detection and configuration. The solution will need to address whether this should be platform independent or platform specific.

Design criteria

Portable

External library support is a potential problem all platforms -- not just Win32 -- and developers will be more inclined to use the solution to specify external dependencies if it addresses dependency problems portably. The solution should provide a standard interface for module developers and module users that abstracts platform and compiler details.

Modular

Library packagers may not wish to take on responsibility for creating a solution for all platforms or all compilers or may lack the necessary know-how. The solution should modularize functionality wherever possible to allow individual library packagers to work on separate parts of the solution for platforms of interest.

User-friendly

The solution should degrade gently. If library support is not available for a particular platform or compiler, the solution should fail cleanly with an explanation of the dependency problem prior to generating XS compilation failures during linking.

Maintainable

Ongoing availability or upgrades of an external library should not depend on a single individual or on tacit, undocumented packaging know-how. As with Perl::Dist::*, the published solution should contain an automated process to convert an external library from source (or binary package) to the binary package format used by the solution.

Design overview and commentary

The following design description will make more sense if you examine the Alien Ecosystem diagram.

Dependency specification

CPAN modules with external library dependencies will specify a module dependency on a corresponding Alien:: module, e.g. Alien::Foo for the "libfoo" module. Dependency management can thus be handled within the existing system using ExtUtils::MakeMaker, Module::Build or Module::Install prerequisite specifications.

Presence of an installed Alien::Foo should be considered evidence that libfoo is available. Alien::Foo must not install if libfoo cannot be found.

Library detection

Library searches will need to be done in platform specific ways. Perl Config values like "libpth" could be used as a starting point.

• Win32: should also include the LIB environment variable and possibly the PATH environment variable for library searches
• Unix: should include the LD_LIBRARY_PATH environment variable (or equivalents); could also use the output of "ldconfig -p"

Library configuration

• Makefile.PL: Alien::Base will provide an API for generating CC flags for both libraries and headers.
• Shared/dynamic library loading: Will need to set the shared library search path before bootstrapping XS. This could be done during import of "use Alien::Foo" or could be done via an explicit class method call.
• Other metadata: Alien::Foo could be extended with methods for library metadata.

Binary library package generation

Alien::Foo::* should include the full procedure needed to generate a binary library package (a ".pal" file) for a particular platform. This should include downloading sources from a URL, any preparation work or patching, and generation of the binary.

This process could be done completely from the upstream library source, or could be a "repackaging" of another binary download, e.g. creating wrappers or definition files around a .dll on Win32.

At worst, this might just be a series of system() calls. The point is to ensure that the full process is available for future packagers to replicate, adapt or enhance.

Alien::Base should provide helper methods for URL downloading, PAL file generation, etc.

After the PAL file is generated, it can be uploaded to a repository for end-users to download for installation.

Binary library package installation

If not available already in platform-standard library directories, the library headers and binaries should be installed into the corresponding "auto" directory for the Alien module. E.g., "site/auto/Alien/Foo" for Alien::Foo. This location should be well-defined for all platforms, since it's the same as is used for XS modules.

Exact layout within this module is still open -- should there be "lib" and "include" subdirectories or should libraries and headers be combined in the top directory. Installation location of binary utilities included with libraries is still an open question.

The binary package may be either generated from source or downloaded from a URL (the default option).

[Corion]

I'm not sure if I have understood your very technical and verbose specification of things, so please bear with me while I rephrase it in my words it as I understand it:

The "PAL" files are to me just a special case of make dist, which I tentatively call make sneakernet, bundling together the external library sources and Perl glue code into a bloated Alien:: module, thus skipping the download, untar and patch steps. Many external libraries are infected with the GPL or other licenses which prohibit bundling with my Alien:: modules, hence this step will have to be performed by the user needing a sneakernet and no aggregate distribution will be provided by me.

What you call Alien::Base is conceptually what my distributions have as inc::Alien::Util, an evolving set of tools that have proven themselves convenient, like code to fetch an URL, code to gunzip/untar a file, code to find the latest version of something on sourceforge.net or elsewhere.

I'm not sure about site/auto - I use $INC{__PACKAGE__} as the base directory below which the C header files and library object files will be stored. In the end, it won't matter as long as it's consistent, of course. Under Win32, additional external DLLs need to go into the same directory as perl.exe, so they can be loaded by Windows without fiddling with $ENV{PATH} and so they don't pollute the "namespace" under C:\WINNT. I prefer the exact layout of include/ and lib/ below the Alien root directory, because that allows most ./configure or other install scripts to remain unchanged and unpatched.

The generation of binary packages to me is completely beyond the scope of the Alien:: namespace - PPMs are the established way of providing binary packages, and in theory, creation of a PPM for package Foo (relying on libfoo via Alien::LibFoo) should automatically pull in the Alien::LibFoo package via nmake ppm and hence produce a fully working Foo PPM, which can then be distributed and downloaded. To me, Alien:: is about automating the process of compiling from source, not about redistributing binaries.

The searching of already installed external libraries through other ways is secondary to me - on Win32, it is almost certain that external libraries are not installed and hence won't be found unless the Alien:: package knows where to download and how to install the prepackaged binary. Even on unixish systems I prefer libraries that have no external life outside of the Perl directory tree so that a system-wide upgrade of a library doesn't affect any Perl program because the Perl programs link to Alien::SomeLib instead of the system-wide library installation, and thus all Perl programs continue to work as before. Of course, an optional switch to make Alien:: scan for pre-existing installations would be nice to have.

The focus in compiling is to me always static linking of the library into the XS file, to remove one level of dynamism from the Jenga tower of shared libraries. This has the added benefit that a manual transplantation of Foo consists of copying all directories below site/Foo and site/auto/Foo, as Foo.xs gets converted into Foo.dll with libfoo linked into Foo.dll statically.

[xdg]

very technical and verbose specification of things

Several of us on #win32 had been speaking very vaguely, so I was trying to break the problem down so individual parts could be addressed discretely. Mea culpa.

I'll add just a few notes in response. I appreciate the thought you've put into it. I definitely need someone with your kind of compiler-fu. As I start taking this towards a proof-of-concept, I'll certainly want to get more of your input.

What you call Alien::Base is conceptually what my distributions have as inc::Alien::Util

Yes, though it would also include tools for interrogating a system to determine if external dependencies were already satisfied or for getting metadata about installed external dependencies.

The "PAL" files are to me just a special case of make dist

It was intended generically. It could be PPM's in the end, or it could be zip/tgz files. I didn't want presuppose a particular solution.

[tsee]

Hi Corion,

thanks for your long comment. As xdg pointed out, we really appreciate your input.

I'd like to address one point: You say that PPMs are the established way of providing binary packages. I think that's true in a way and false in another. It's the binary package format that is established with ActiveState Perl.
I know they just rewrote all of the PPM code and supposedly created a really shiny GUI. (I have only seen a couple of screenshots so far.) That's great, but if I am not mistaken, we are not free to use their PPM code with non AS Perl distributions. At least I could not find it on CPAN. This means that if we're building something for any Perl distribution to use, we shouldn't lock ourselves into the PPM cabinet. Do you agree?

Steffen

[Corion]

I don't agree in the sense that PPM is an established, working system, and producing PPM files needs no PPM code that isn't available already in Module::Build and I guess ExtUtils::MakeMaker. I'm not interested in creating yet another fancy incompatible format and pushing it down users throats, or rather, I'm more under the impression that there won't be users with throats open for this product. I'm not interested in creating a shiny new world order external library packaging factory but more interested in producing working modules that install external libraries via CPAN below Perl, provided that $Config{make} and a working $Config{cc} are available.

[tsee]

Just a quick update for the interested:

Instead of "site/auto/Alien/Foo" for Alien::Foo, the lib target will most likely be "site/auto/Alien/lib". While that's not great because it might introduce file name clashes, the length of environment variables such as PATH and LIB is severely limited by the operating system. Hence, freely adding an entry at runtime for every Alien::* module loaded might be a very bad thing.

Steffen

[Anonymous Monk]

Next time maybe you can link to the doc instead of pasting the whole thing?