On packaging modules

[nothingmuch]

Howdy..

I've been doing two things, recently:

• packaging stuff for CPAN
• cpansmoking, specificallly with older perls

and I'd like to share some of what i've learned.

Update: Looking at this again, I'd like to stress that this is not a tutorial or a howto on packaging. It's a number issues that I consider either important or beneficial when incorperated into your packaging workflow.

The tools

This is not really a step into packaging modules correctly, but a prerequisite. Use Module::Build. It's shiny, it's sexy and it's future proof. It's also backwards compatible, with traditional Makefile.PL generation. If you have a reason, use ExtUtils::MakeMaker instead.

Update: Both Zaxo and Corion expressed concern about my advocation of Module::Build. They explain a problem with the faked Makefile.PL interface to it (which IMHO is sillly - if you make a Makefile at least remove the dependancy from M::B).

Anyway, I sort of take it back. Read their posts, see what they have to say. I bought M::B's story, but I may be an idiot. Ciao!
</Update>

From here on, unless noted, instances of ./Build or Build.PL are canonical to, and can be replaced by make or Makefile.PL if you went the ExtUtils::MakeMaker way. Keep in mind that Michael Schwern, MakeMaker's maintainer promotes replacing MakeMaker with Module::Build.

What goes in

The first step of properly packaging a module has to do with what goes in, and what stays out.

A confusing aspect of that is that what goes into your src dir is not necessarily what goes into your tarball. This is where we'll start.

Enter ./Build dist

Both Module::Build and ExtUtils::MakeMaker provide a tool to construct a properly compressed, well defined, signed, manifested, META.yml'd tar ball for you, based on your MANIFEST file.

Given a proper MANIFEST, run perl Build.PL and then ./Build dist. If you want a SIGNATURE file generated, put the argument sign => 1, in your Build.pl args. If you want a Makefile.PL, put a create_makefile_pl => 'traditional', in there too (but make sure Makefile.PL is mentioned in your MANIFEST!).

MANIFEST

So, what goes into the dist is really what's in the MANIFEST, right? Do we really want to edit that? No.

Instead, write a MANIFEST.SKIP file. My generic one looks like this:

\.DS_Store$
^_build
^Build$
^blib
~$
\.bak$
\.sw.$
^cover_db
^MANIFEST\.SKIP$

The first line removes the pesky OSX .DS_Store stuff. The next 3 lines take care of skipping build cruft. Any editor backup files are skipped, as are vim swap files. Devel::Cover dirs are also ommitted, and lastly, so is the MANIFEST.SKIP file itself. I sometimes add \.tar\.gz$ to the mess, to make sure that dists are not accidentally added to the manifest too.

Now that we defined what we want out of the manifest, lets get our tool to generate one for us: perl Build.pl; ./Build manifest And now, any file that doesn't match the patterns in MANIFEST.SKIP is mentioned in there.

General Kwalitee

After we define what goes in, we have to make sure it isn't garbage.

Respect your elders

Lets start with the bare essentials. Here is an example Build.PL:

#!/usr/bin/perl

use strict;
use warnings;

use Module::Build;

Module::Build->new(
        module_name => 'My::Module',
        license => 'perl',
        requires => {
                'Other::Module' => '0',
        },
        build_requires => {
                'Test::Funky' => 0,
        },
        create_makefile_pl => 'traditional',
        sign => 1,
)->create_build_script;

That looks well behaved, right? Lets see what's wrong with it:

• use warnings; - warnings.pm is only available from 5.6.0. Lets remove it, and put -w on the shebang line instead. It's very frustrating to have to figure out something is broken from 30 lines of error messages. A 'sorry, this perl is xxx, i need yyy' message is much more polite.
• requires - it's a bit empty. What's our minimum perl? What about core modules? Test::Prereq will help you take care of this. Lets say this module can run from perl 5.6, but needs Scalar::Util. This will cause ugly build failures later on.
• build_requires - this is also quite lax. Test::More was not core in older perls. I don't think i've seen a Test::Harness friendly suite that doesn't use it.

Basic sanity

Test::Prereq was mentioned earlier. It has some siblings. You can write tests which make sure your packaging, and not only your code, is good.

• Test::Distribution - this module does it all. See below
• Test::Prereq - tests that your prerequisites are in order.
• Test::Pod - looks for errors in pod files.
• Test::Signature - makes sure your SIGNATURE file is cryptographically valid.

With these tools, you can have your code check for you, again and again, that you are not uploading junk.

It should be noted that Test::Prereq might not apply to modules you optionally use in tests. skip_all is often used when these modules aren't available. Test::WithoutModule helps you check if that code works correctly.

Tests

Now that we've touched packaging quality of tests, are you writing real tests too? These are more important you know.

cpansmoke will complain if you include no t/ dir. It will send you an email saying "would you please get up off your ass and at least write a tests that does use_ok("Your::Module")".

This has become an integral part of a proper distribution, even if it doesn't really do any real testing. At least try to look like you're trying.

Versioning

Module::Build will look in the lib/ dir for a file that implements the module you named in the Build.PL script.

From there the value of $VERSION is extracted, and used as the distribution version.

Versions are important because they imply compatibility, they define dependancies, and so forth. On the CPAN there is a general format.

• \d\.\d (0.01) - this is the standard released version. The major version number is usually incremented when an interface drastically changed, and the minor number when something is fixed. Sometime fixes are incompatible. The 3 number versioning scheme, which is not very standard on CPAN usually implies that the third number is compatible fix, and the second number implies some interface change. Try to be consistent with these ideas when versioning, for both your users, and your depending modules.
• \d\.\d+_\d+ (0.01_02) - This is how you denote a developer release. Developer releases are not downloaded and installed unless explicitly asked for by the various managers. The http://search.cpan.org interface warns about these too. If you want to have your code available and tested, but it isn't quite ready, use this format.
• \d\.\d+a-b (0.01a) - This is like the previous developer release format. Letters denote developer releases.

Summary: Source tree overview

Your tarball, when finished, should make sense. Here is what a proper one looks like, with all the necessariy, desired, and reccommended files in place.

Your-Module-0.01/
|-- Build.PL # your build script.
|-- LICENSE # put you're license here if you really feel like it
|-- MANIFEST # we discussed it already
|-- Makefile.PL # this too. For those who aren't using CPANPLUS yet.
|-- Meta.yml # this is generated by ./Build dist
|-- README # everyone should have one
|-- SIGNATURE # like MANIFEST, only cryptographic
|-- lib/ # this should look like something you could put in @INC
|-- t/ # this is where you keep your tests
|   |-- lib/ # if your tests need some libraries, put them here, and u
+se lib 't/lib'

And your source dir should be:

src/
|-- Build.PL
|-- LICENSE
|-- README
|-- lib/
|-- t/
    |-- lib/

Build dist and friends should also generate:

src/
|-- MANIFEST
|-- Makefile.PL
|-- Meta.yml

And ofcourse, there is cruft:

src/
|-- Build # left over from perl Build.PL
|-- Makefile # or perl Makefile.PL
|-- _build # where Build.PL keeps it's data
|-- blib # where your source tree is before installation
|-- *~ # your editor backups
|-- cover_db # your data from test coverage
|-- CVS # your source control meta data

That last part is there, but you shouldn't really care about it. Manipulate your MANIFEST.SKIP if they bother you.

References

Module::Build is the tool to use for your package making needs.

Test:: is a namespace you should know. See also Test::Distribution and it's friends.

Devel:: is a namespace to help you write code. Deve::Cover, for example, is a away to see what code is getting run when you run tests.

Module::Signature makes cryptographically signed MANIFEST like files.

The CPAN testing service is an experiement to try and measure package quality.

http://testers.cpan.org run tests for you, on many platforms and perls, when you release modules with tests.

http://qa.perl.org is a website concerned with the quality of perl and CPAN in general. It's an interesting starting point.

Addendum

I don't believe in h2xs. It's yucky. Test::Tutorial has a nicer way to start out. Aside from that, all that's important is that you have the proper sections in your pods. Test::Pod, and podchecker will both help you with that.

That doesn't mean you shouldn't see what h2xs is outputting, and understand what the mess it makes is.

P.S.
my brain is a bit zonked today. This probably has many grammer errors, ambiguities, and disinformation. Please inform me if you find any!

[Zaxo]

An attempt to install Module::Build over CPAN gives:

  CPAN.pm: Going to build K/KW/KWILLIAMS/Module-Build-0.2604.tar.gz

Sorry, PREFIX is not supported.  See the Module::Build
documentation for 'destdir' or 'install_base' instead.

Prior attempts over the last year have all failed for one reason or another. I'm reluctant to trust a builder which cannot build itself, and is so dismissive of current practice as to knowingly ignore PREFIX.

The objections to make are mostly avoidable by keeping things simple, only using POSIX features, writing for /bin/sh. Everybody needs make, anyhow, whether they know it or not ;-)

[kappa]

Module::Build::Cookbook says that "newer versions of CPAN.pm support building via ./Build.PL". Alas, I could not find that in the source. Strange. FreeBSD Ports e.g. support Module::Build for a long time.

[itub]

I'm glad to see that I'm not the only one who is extremely annoyed by Module::Build's disregard towards PREFIX . The Module::Build documentation says:

Note that this is different from how MakeMaker's PREFIX parameter works. PREFIX tries to create a mini-replica of a site-style installation under the directory you specify, which is not always possible (and the results are not always pretty in this case).

Ok, so they decided not to support PREFIX because they don't think it's pretty? I don't know what they mean when they say "it is not always possible". PREFIX has always worked for me (with ExtUtils::MakeMaker), and I assume that other people who choose to use it also found that it works for them.

[dragonchild]

More importantly, why isn't it always possible (or pretty)? If it's because the C-64 or the Amiga doesn't support it, I couldn't care less. But, frankly, if s/he explained the problem, maybe a few more heads would find a solution to the problem ...

[lachoy]

Jeez, give them a little credit would you? Read more about the nastiness of PREFIX, and take a breather before getting so annoyed next time :-)

[ysth]

There's been discussion of this issue on the module-build-general list recently and there may be some movement on this front. Check back in a month or so.

[Anonymous Monk]

Zaxo wrote:

The objections to make are mostly avoidable by keeping things simple, only using POSIX features, writing for /bin/sh. Everybody needs make, anyhow, whether they know it or not ;-)

I really, really, really hope that's a bad joke.

This might come as a shock to you but not everybody runs Unix. Even if they did, not all Unixen are the same. Even if they were, not all makes are the same. There's BSD make, GNU make, Solaris make... Even if they were the same, they have bugs and not everyone uses the latest version. Even if they did there's different compilers, different build tools, different file tools, different file systems... oh the incompatibilities just keep coming.

Trust me. I have to deal with all of them. I maintain MakeMaker. And MakeMaker has to work everywhere Perl does. Be glad it does. It would suck if you couldn't install Perl modules on your OS because some unrelated utility is quirky. If working with MakeMaker has taught me one thing its not to be such a bloody Unix snob!

Even if everyone used the same Unix variant on the same hardware with the same linkers, compilers and utilities and there were no bugs, make would still be the problem. It has always been the problem. Not just make but relying on any external build tool. I have learned this after many painful years of trying to clean up MakeMaker. It is impossible. I'm going to let you read why because I've said all this so many times before that I've written a whole talk about it.

Its taken years and 1400+ lines of code to keep MakeMaker running on VMS (about as far from Unix as you can get). I ported Module::Build over in one night and 25 lines.

[Corion]

This is not really a step into packaging modules correctly, but a prerequisite. Use Module::Build. It's shiny, it's sexy and it's future proof. It's also backwards compatible, with traditional Makefile.PL generation too. If you have a reason, use ExtUtils::MakeMaker instead.

Module::Build is not backwards compatible, as it doesn't respect the PREFIX= syntax and provides no means to achieve a similar result. So please avoid Module::Build unless you have compelling reasons to use it. ExtUtils::MakeMaker is the default, standard and working solution. If you think you really need Module::Build, create a fully compatible Makefile.PL too.

[nothingmuch]

With the traditional makefile, you get a Makefile.PL that works with ExtUtils::MakeMaker. I personally see this as enough.

As for the other arguments against Module::Build - I will edit my post.

[Anonymous Monk]

For those of you who don't know, I'm the poor sod who has to maintain ExtUtils::MakeMaker, as somebody here put it "the default, standard and working solution". Ha.

First of all, MakeMaker is DOOMED! Its rotting from the inside out, the architecture is all wrong. Has been since the beginning . It would take more effort and cause more breakage to properly fix it than to just write a whole new build system. ExtUtils::MakeMaker has to go away. Not today. Not tomorrow. But it has to happen. Module::Build is the heir apparent. I'm helping out as much as I can with it.

Second, PREFIX does not really work. Just because it works for you and your configuration of Perl does not mean it works for everyone else's. And that's the game build systems have to play in a language as ubiquitous as Perl. It has to work. Not only that but it has to work everywhere! And I'm not just talking about VMS or MacOS , I'm talking about the hundreds of Unix variants out there which all have their own way of configuring Perl. I started maintaining MakeMaker because PREFIX was broken. The system I was using was Debian.

Read more... (1341 Bytes)

What is needed is a simple, predictable, user configurable alternative. Module::Build's install_base is the beginnings of that. It installs into the same layout no matter what your Perl configuration. The layout itself is still being tweaked (for example, it currently goes into foo/lib. It should probably be foo/lib/perl5) but its pretty much ok. It is customizable by passing in more install* arguments to Build.PL.

The future is install_base. MakeMaker will be implementing its own INSTALL_BASE logic to match.

What's missing is an easy way to customize your layout once and then be done with it. Module::Build will likely support a .modulebuildrc file where you can write out your defaults and be done with it.

As a final note, Module::Build's compatible Makefile.PL will be getting PREFIX support. At the OSCON 2004 auction I offered to implement PREFIX for Module::Build if a few hundred dollars were donated to TPF. Folks ponied up the money (alas, I do not have their names) and it will be done.

[tye]

Now if we can get Module::Build to include a pass-through Makefile.PL by default, then we'll have eliminated the major reason a ton of people have sworn off Module::Build (other than the impression of personality problems that lead to broken decisions like this).

(I'm assuming that the major Win32-related issues with Module::Build have been addressed.)

There will certainly be other issues to address, but this particular one needs to be addressed immediately.

The second issue will probably be getting Module::Build into core Perl.

[schwern]

Now if we can get Module::Build to include a pass-through Makefile.PL by default

No, Module::Build should not produce a Makefile.PL by default. MakeMaker can't do all the things Module::Build can do and vice-versa. The generated Makefile.PL will not contain any extra logic in the Build.PL, such as asking the user configuration questions or sniffing the environment.

But most importantly, MAKEMAKER HAS TO DIE!

Having Module::Build attempt to be backward compatible with MakeMaker is a temporary solution while MB firms up and CPAN utilities catch up with it (CPANPLUS 0.50 finally handles MB pretty good). It has to take over fully from MakeMaker. Ubiquitous Makefile.PLs just delay this.

I'm assuming that the major Win32-related issues with Module::Build have been addressed.

Something projects like Module::Build and MakeMaker are always short on are Windows developers and Windows machines to check stuff out on. I can almost never find someone to check out MakeMaker releases, its actually easier for me to find VMS folks than Windows folks. And at the very least I can log into a remote VMS machine and test things out. I don't have any such thing for Windows (VNC does not count, its not multi-user AFAIK).

This is not a dig at Windows, its a problem statement. If you know of a way we can log into a remote Windows server to test things out, let us know. Or if you know some font of competant Windows Perl developers send them over to [e-mail] and [e-mail].

There's a bunch of open Win32 bugs on rt.cpan.org. Have at 'em.

The second issue will probably be getting Module::Build into core Perl.

MB is slated to go into 5.10 along with CPANPLUS.

[dragonchild]

src/ directory? I do my editing in lib/ and then do a make tardist when I'm ready to go. What does a src/ directory get me?

[ysth]

Sounds like the src/ directory is just wherever you happen to keep the module you are working on. Don't think it was intended to be something actually in the distribution.

[nothingmuch]

Uh, yeah, what ysth said. It's the hypothetical working directory (note that it's not in the tarball listing), where you keep your build system script, Changes file, and so forth.

It's what contains lib.

I tried posting twice, appearantly I was too much of an airhead yesterday to go past the preview stage...

Ciao!

[petdance]

Module::Starter is your current best choice for creating a distro from scratch.