Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Making Inline::C modules fit for CPAN

by tsee (Curate)
on Nov 15, 2006 at 08:53 UTC ( #584125=perlmeditation: print w/ replies, xml ) Need Help??

Update: I have since release InlineX::XS to CPAN which is a more robust implementation of the ideas outlined below.

There are a couple of modules on CPAN that use Inline::C. If you don't know what Inline does, you should go read up on it now because for the programmer, it is by far the simplest and most convenient way to write fast C extensions to your Perl code.

In order to hide as much of the compilation process from the user, Inline::C automatically generates XS from the inlined code, compiles it, and links it into the running perl. It will use a dedicated cache area for that and recompile whenever the code was changed. Unfortunately, this means that unless you ship the cache alongside your module or application, every user requires a full C development environment and the perl headers. Of course, for building the application or module in the first place, you need that anyway, but there is a specific, very important example where users don't have that: They install modules using PPM or PAR. PPM and PAR packages are binaries created from CPAN modules. In case of ordinary XS modules, these packages can install all required dll's/so's. This is a hassle with Inline::C-using modules.

Some authors convert their modules from using Inline::C to ordinary XS for this reason. There is even a utility module on CPAN which helps with this: Inline::C2XS can write an XS file from the (plain) C code of an Inline::C-using module. But this conversion is manual and as such, a maintenance nightmare.

I'm now going to propose a possible way out of this dilemma. We want to let the module author have the comfort of using Inline::C for his module and the user should be able to install the module as an XS extension. Additionally, the author should not have to manually extract the C code from his module in order to make a release.

Here's the concept: Say, we're dealing with a module Foo::Bar which contains inlined C code. The trick is to auto-generate an XS file from this C code during the make dist phase and subsequently ship that XS file with the module. During the make phase on the user's machine, the XS file is compiled to an ordinary XS extension. If the associated DLL/so is present, Foo::Bar will not use Inline to compile the inlined C code, but it will use XSLoader to load the existing DLL. This way, the author can use Inline::C for development and when ready, ship the module as an ordinary XS module. Enter the hypothetical Module::Inline::C.

  • Module author uses Module::Inline::C whereever he previously used the normal Inline::C.
  • When Foo::Bar is used, the C code is passed to the import routine of Module::Inline::C which subsequently decides what to do with it:
    1. If it can load the associated XS-dll, it loads that and discards the plain code.
    2. If that fails, it transparently passes everything through to Inline::C using a few hacks.
  • During make dist, it would have to be arranged that Foo::Bar is compiled once with a specific command like:
    perl -c -MModule::Inline::C=PACKAGE blib/lib/Foo/Bar.pm # (or similar)
    That way, perl would load Module::Inline::C first and tell it that it should write XS in this execution. Then, when perl compiles the use() statements in Foo::Bar, it executes Module::Inline::C's import() routine which writes the C code to an C file in the distribution. At the end of the compilation phase, Module::Inline::C invokes Inline::C2XS to generate an XS file which is fit for inclusion in a CPAN distribution.

Now, this whole scheme should sound reasonably scary to you or else you're just plain sick. That's why I wrote some proof of concept code to show how this might work:

Foo::Bar

package Foo::Bar; use 5.006; use strict; use warnings; our $VERSION = '0.01'; use Module::Inline::C <<'HERE'; int fac (int x) { if (x <= 0) return(1); return(x*fac(x-1)); } HERE # ordinary perl code and more inlined C here # This is for triggering the XS generation if applicable: use Module::Inline::C 'END'; 1; __END__ =head1 NAME Foo::Bar - Perl extension for blah blah blah =cut

Module::Inline::C

package Module::Inline::C; use strict; use warnings; # Should work for *one Inline::C-using package per distribution only* # right now! our @INLINE_ARGS; our $PACKAGE = 0; sub import { my $class = shift; my @args = @_; if (@args==1 and $args[0] eq 'PACKAGE') { warn 'Setting PACKAGE option'; # trigger packaging/XS generation mode $PACKAGE = 1; return 1; } elsif (@args == 1 and $args[0] eq 'END') { # All C code was received. Generate XS. return unless $PACKAGE; _generate(); } elsif ($PACKAGE == 1) { warn 'Saving arguments to Inline because we\'re in PACKAGE mod +e'; # Write out C code my ($pkg) = caller(0); push @INLINE_ARGS, {pkg => $pkg, args => \@args}; } else { # try to load dll/so first (user mode) warn 'Trying to load shared obj file'; my ($pkg) = caller(0); require XSLoader; eval { XSLoader::load($pkg); }; return 1 if not $@; # Compile using Inline::C (author mode) warn 'failed to load shared obj file, resorting to inline'; eval "package $pkg; require Inline; Inline->import('C', \@args +);"; die $@ if $@; return 1; } } sub _generate { require File::Spec; require Inline::C2XS; mkdir('src'); foreach my $call (@INLINE_ARGS) { my $pkg = $call->{pkg}; if (@{$call->{args}} != 1) { require Data::Dumper; warn "Skipping Inline C call from package $pkg with argume +nts: ".Dumper($call->{args}); next; } my $file = $pkg; $file =~ s/^(?:[^:]*::)*([^:]+)$/$1/; $file .= '.c'; open my $fh, '>>', File::Spec->catfile('src', $file) or die $! +; print $fh "\n".$call->{args}[0]; close $fh; Inline::C2XS::c2xs($pkg, $pkg); } } 1;

There are various issues with the code as it stands, but the general idea seems reasonably sound: Let the author write maintainable inlined code and let the user have ordinary XS extensions. I'm just going to point out one prominent omission: Module::Inline::C (bad name, I know) doesn't keep track of multiple packages using it at the same time. But that's just a bit of house keeping!

What do you think? If I hacked this up to a reasonably stable state and put it on CPAN, would you consider using it for your C extensions?

Cheers,
Steffen

PS: Yes, I actually ran this. It works for me.

Comment on Making Inline::C modules fit for CPAN
Select or Download Code
Re: Making Inline::C modules fit for CPAN
by rinceWind (Monsignor) on Nov 15, 2006 at 12:19 UTC

    tsee++

    Well done, Steffen for the sterling work you are doing with this, and the recent developments with PAR.

    Although I have a C background, I've been too scared to date, to get my hands dirty with coding XS, and at least one of my CPAN modules would benefit from some C routines. I've also been scared of Inline::C and the voodoo involved in getting it working with CPAN and package managers, so I welcome your insights.

    I'm wondering if it's a design defect of Inline::C that it needs to runtime compile the code. This is making a big assumption, that your target machine has a working C compiler. I'm very interested in your workaround, especially one that will work with CPAN and package managers.

    Please also consider all the package managers that are out there: e.g. from the Linux world - dpkg, rpm and evolve (and others).

    --

    Oh Lord, won’t you burn me a Knoppix CD ?
    My friends all rate Windows, I must disagree.
    Your powers of persuasion will set them all free,
    So oh Lord, won’t you burn me a Knoppix CD ?
    (Missquoting Janis Joplin)

      I'm wondering if it's a design defect of Inline::C that it needs to runtime compile the code

      I don't really see it that way. It only needs to "runtime compile the code" the first time the code is run. Subsequent runnings of the code simply load what was compiled the first time ... no need to re-compile it. In that way it's no different to a perl extension - ie it needs to be compiled only once.

      This is making a big assumption, that your target machine has a working C compiler

      This statement implies to me that you're thinking Inline::C will work without a C compiler - which is not the case. (Sorry ... perhaps you meant something entirely different.)

      Cheers,
      Rob

        I think you've not appreciated my point, though I admit I probably didn't express it very well.

        A CPAN style distribution has a 4 phase install process: configure, build, test, install.

        perl Makefile.PL # Configure make # build make test # test make install # install

        or

        perl Build.PL # Configure ./Build # build ./Build test # test ./Build install # install

        For an XS or swig module, compilation of C code takes place in the build phase. This phase generates usable deliverables: perl code, binaries, man pages, scripts, etc. under the blib directory. The test harness mechanism works on these deliverables pre-installation, so you can run tests against your distribution before it gets installed.

        Please correct me if I'm wrong in any of this - this is based on my understanding of I::C and I could be talking rubbish here. An Inline::C module won't be compiled during the build phase, but the first time it is run. While this should (hopefully!) happen as a side effect of the test phase, these binaries won't be listed as deliverables, and won't be installed as part of the install phase. So, the binaries get built, the first time the module gets called after it has been installed.

        This won't happen as root, most of the time, so presumably the compile is redone for each user, the first time they run the application that uses the script. If this isn't the case, there could be permissions problems and all sorts of environmental wierdness happening.

        So far, I have been referring to the CPAN model for installing a distribution. When it comes to package managers, the idea (usually) is that there is no compile step done on the target machine. Said target machine could be Windows with a PPM install, Debian with an install via apt-get or whatever. The point is that package managers assume that no compile is needed on the target, and no compiler is necessary. This implies that distributions for such package managers may be platform specific and contain pre-built binaries.

        The important distinction here is that the build machine has a C compiler, indeed the full development tool set. But the target machine is not guaranteed to have any of this, it's a run-time environment.

        PAR takes this idea a stage further, and the target machine doesn't even need to have perl installed.

        Hopefully this has clarified my argument sufficiently.

        --

        Oh Lord, won’t you burn me a Knoppix CD ?
        My friends all rate Windows, I must disagree.
        Your powers of persuasion will set them all free,
        So oh Lord, won’t you burn me a Knoppix CD ?
        (Missquoting Janis Joplin)

Re: Making Inline::C modules fit for CPAN
by tsee (Curate) on Nov 16, 2006 at 17:56 UTC

    Okay, I got a little further. Along the way, various issues of varying potential to make me really sick came up. Supposing I release Module::Inline::C to CPAN, here's the recipe for converting your Inline::C-based CPAN distribution to use Module::Inline::C. This works for MakeMaker-based distributions only at this point.

    • Replace all invocations of Inline with M::I::C:
      # use Inline C => '<<HERE'; # becomes use Module::Inline::C <<'HERE'; ... HERE
    • Replace the 1; that ends the module/.pm file with use Module::Inline::C 'END';
    • The version declaration of your module needs to be accessible at compile time and also determinable by MakeMaker which uses a simple regex to extract it. That means you need to do this:
      # our $VERSION = '0.01'; # becomes our $VERSION = '0.01'; BEGIN {$VERSION = '0.01'};
      (Similarly if you prefer use vars '$VERSION'.)
    • You need to add a bit to your Makefile.PL. For module Foo::Bar, this might look like:
      use ExtUtils::MakeMaker; my $module_file = 'lib/Foo/Bar.pm'; WriteMakefile( NAME => 'Foo::Bar', VERSION_FROM => $module_file, PREREQ_PM => { 'Module::Inline::C' => '0', # instead of Inline }, ($] >= 5.005 ? (ABSTRACT_FROM => $module_file, AUTHOR => 'Your Name <and@e.mail>') : ()), dist => { PREOP => "perl -MModule::Inline::C::MM=\$(DISTNAME)-\$(VERSION +) -c $module_file", }, );
      The dist => ... part specifies that before tar-ing up the distribution file, the Inline::C to XS conversion should take place. The MANIFEST of the generated distribution is updated automatically.

    That should be mostly it. Afterwards, anybody downloading and installing your distribution should be able to do the usual

    perl Makefile.PL make # compiles XS => so/dll into blib/ make test # doesn't use Inline! make install # no more Inline for this module, ever.

    Of course, the user is free to delete the .xs files from the distribution and use Inline instead. Using Inline::C can be much more enjoyable for development (IMHO).

    There's still a lot of cleaning up to do as well as adding documentation and implementations for Module::Build and Module::Install. Additionally, if you have a distribution with several XS (or rather: Inline::C) modules, you might run into trouble with this. But you're welcome to read up on the issue. It's not specific to this but rather common to all XS distributions.

    Steffen

    Update: I forgot one item in the list.

Re: Making Inline::C modules fit for CPAN
by tsee (Curate) on Nov 17, 2006 at 17:38 UTC

    A first cut of a complete Inline::XS (formerly Module::Inline::C) implementation is ready.

    I haven't uploaded it to CPAN yet since I want the inline community's blessing for the namespace choice before doing that. Until it hits CPAN, you can grab a copy of it from my website. The link might go down as soon as the real thing hits CPAN.

    Steffen

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://584125]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (12)
As of 2014-10-21 14:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (105 votes), past polls