Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Do Pure Perl CPAN packages really need to use ExtUtils::Command::MM?

by ELISHEVA (Prior)
on Feb 17, 2011 at 16:37 UTC ( [id://888753]=perlquestion: print w/replies, xml ) Need Help??

ELISHEVA has asked for the wisdom of the Perl Monks concerning the following question:

CPAN has two installers, the more "modern" Build.PL backed by Module::Build and the "traditional" Makefile.PL backed by ExtUtils::MakeMaker and ExtUtils::Command::MM. The convention for CPAN modules is to ship a package that supports both the old and the new installers.

The problem I'm running into is this: it is impossible to honor the conventions of both the older and the newer installer at once. When you have libraries that are used by the build/test process but not part of the runtime code, the newer installer (Module::Build) expects you to put them in a subdirectory of the project root, preferably named inc. The older build process expects you to put everything in the project root. This is hardcoded into the software and cannot be changed.

The newer one is more flexible and can accommodate the older conventions, so if I want to support both tools using their native build command processor, I have to go with the older conventions. I like the newer conventions. They seem cleaner and facilitate automation.

The older conventions make automating and customizing the packaging process much more complex. Instead of being able to say "everything in inc is mine", any automation software I write has to scan the root and decide what is mine, and what is generated by the build process. That means a fragile hard coded list of what other products I don't control are liable to put in root or else an easy to forget to maintain list of what I put into root. But even if it weren't fragile, any automation that has to dance around exclusion lists like that is more time consuming to write.

There is a third option suggested in the cb last night: place a use lib 'inc'; statement in any file that needs to call support files. If I did that I could do things the Module::Build way but still use ExtUtils::Command::MM. It is a tried and true method and no less than Michael Schwern uses it. (Note: he's the author of the older installer).

I'm probably sounding like Sam in Green Eggs and Ham right now, but I don't like that idea either. First, it means that I have to go into every file I want to ship and edit it to have that line.(Source filters are out - I don't trust them). Secondly, these support libraries are used in a lot of different projects - sometimes at runtime and sometimes at build/configure. If I have any concerns about the current directory of the caller, I also need to add Find::Bin which doesn't work under mod_perl. Finally, as a general rule, I don't like to manipulate @INC in .pm files. That job should be left to calling scripts and configuration files. Separation of concerns and all that. It marries assumptions about file locations with source code that could be used in any number of projects and in any possible role (runtime or support). Also any configuration that requires going into each and every file is bound to be more time consuming than a configuration that doesn't.

I suppose you could say, well, what's it to you: how many files are we talking about? Why do you even need automation? To which I'd counter, does it matter? The more time I spend hand-crafting each project, the fewer projects I can manage at one time. The answers to the questions are interdependent. Philosophically, it bothers me as well: this smells of building to the tool. My TMTOWTDI sense doesn't like it very much at all. Green eggs at least tasted good in the end. This doesn't.

This leaves me with two options: (a) bite the bullet and accept the older conventions. (b) take advantage of a feature of Module::Build that generates a traditional Makefile.PL with a twist. Its build targets simply pass commands onto Module::Build rather than the ExtUtils::Command::MM module used by the older build process.

The only downside I can see to option B is that those using very old legacy perl interpreters might not be able to install my CPAN modules (Module::Build would then become a prereq). Module::Build joined the Perl core with release 5.9 started in October 2003. It became part of stable 5.10 in Dec, 2007, over four years ago.

But is four years old, really all that old? On the timescale of new software fads and mobile phone tradeins, four years is ancient history. In the time scale of much more conservative back-end processing - the place where Perl thrives - it is not that old at all.

I don't know if the people testing my CPAN software are in any way representative, but only 3% of testers were even trying to run the older installer. 97% were using Module::Build. Is it really worth marrying my code to something I consider ugly and hard to maintain and anti-automation just to satisfy an increasingly legacy need? What do you think?

Update 1:Eliya suggested a possible resolution to the differing approaches used by ExtUtils::MakeMaker and Module::Build. Using post_initialize you can add directories to PERL5LIB and get extra paths into @INC that way. With the help of File::Spec and Config this can be done in a portable way. No conflict, no need for option B below. Yeah! - see Re^3: Do Pure Perl CPAN packages really need to use ExtUtils::Command::MM?

Update 2: An anonymous monk reminds me that the portability questions aren't all answered - see Re^7: Do Pure Perl CPAN packages really need to use ExtUtils::Command::MM?. Also moved the updates to the bottom because there still isn't a conclusive answer. At present the options are between Option B above and eliya's solution below.

Replies are listed 'Best First'.
Re: Do Pure Perl CPAN packages really need to use ExtUtils::Command::MM?
by jethro (Monsignor) on Feb 17, 2011 at 17:48 UTC

    Comments:

    * There are still fans of MakeMaker out there, for example Matt S. Trout was defending it in a talk I heard last year. Maybe it is not so legacy as you think

    * What is the sense of two installers if everyone has to support both?

    * Machines where old perl versions operate usually don't get much new software. But if the maintainers of those machines are allowed to install your modules, why shouldn't they be allowed to also install Module::Build (which "works fine on perl 5.005" according to the docu) ?

    So I would recommend going with (b)

      it works fine on perl 5.005 if you can install a few additional modules
      If that is the case, then why does Module::Build require a Perl of at least 5.6.1? The Changes file says that requirement was added in 2008.

        I couldn't find a 'require version' or 'use version' line in the source code of Module::Build. There is only that single line in the changelog. You might ask schwern about it or send the maintainers a bug report. The documentation only says that since 5.6.0 all prerequisites are included in core, but works fine on 5.005.

        Anyway, does that change much? Companies that still use a pre-5.6.0 versions are usually so conservative that they don't allow any modules from CPAN anyway

Re: Do Pure Perl CPAN packages really need to use ExtUtils::Command::MM?
by ikegami (Patriarch) on Feb 17, 2011 at 18:14 UTC

    Your older vs newer tags are wrong — there are at least three well used installers — and you surely use them to attempt convey false impressions of unfitness. Don't do that.

    The older build process expects you to put everything in the project root. This is hardcoded into the software and cannot be changed.

    Nonsense. MM doesn't care one bit. It has no expectations at all in this area.

    (Upd: By default, it would install .pm found in the room directory. One could say it expects you not to put test libraries there, the opposite of what you said. )

    There is a third option suggested in the cb last night: place a use lib 'inc'

    Just like you would have to do with any other script that wants to use uninstalled modules, the script needs to tell Perl where to find the modules.

    Your syntax is wrong, though. It makes assumption about the current directory. Running the test from the directory in which its located would fail.

    I don't know if the people testing my CPAN software are in any way representative, but only 3% of testers were even trying to run the older installer. 97% were using Module::Build.

    That's completely false. I imagine that 100% of testers try to use Makefile.PL, not 3%. They just might not try it first.

    (Upd: Furthermore, that shouldn't be taken to mean Build.PL is preferred. It makes more sense to try Build.PL first because users of Build.PL often provide a Makefile.PL for compatibility. )

      Nonsense. MM doesn't care one bit....

      I think we are misunderstanding each other. MM certainly isn't removing things from @INC or dying if it sees something it doesn't like. I am referring to the contents of @INC before any script calls use lib or any command line sets and exports PERL5LIB.

      The contents of that @INC are hard coded to include the usual perl directories + the root of the project + blib/lib + blib/arch. MM generates the "test" target in the makefile using sub test in ExtUtils::MM_Unix (or whatever OS) which inherits from ExtUtils::MM_Any. The generated line in the make file is:

      PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-MExtUtils::Command::MM" "-e" "test +_ha rness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES +)

      ExtUtils::Command::MM::test_harness adds $(INST_LIB), i.e. blib/lib, and $(INST_ARCHLIB), i.e. blib/arch to @INC. It has no provision for adding anything more to @INC short of completely overriding sub test and generating my own custom test target. I call that hard coding.

      I simply don't buy that it is a "good thing" to go around coding "use lib" statements in every single .t file. Module::Build does not make me do it. It gives me a way to make a one line change that automatically ensures that every test and every script knows where to find its support libraries without a single "use lib" statement. Do you really believe the world is a better place if I have to add a line to 16 files rather than one line to one file?

      Just like you would have to do with any other script that wants to use uninstalled modules

      Um no. Not just like I would do with any other script. To begin with, some of the modules I am thinking of distributing on CPAN get used in other projects that have nothing to do with CPAN (yes I own the IP). I have no need for that "use lib" statement when I run tests on those same modules for those other projects. I have no need of it when I let Module::Build build the project. I only have need of it, if I need it at all, to accommodate MM. I call that building to the tool. I also call that violation of separation of concerns.

      I imagine that 100% of testers try to use Makefile.PL, not 3%. They just might not try it first.

      I am not proposing doing away with Makefile.PL. Option B keeps Makefile.PL but simply implements the targets with Module::Build rather than ExtUtil::Command::MM. You can still use Makefile.PL to generate a makefile and have it work perfectly if you feel more comfortable that way.

      You might well be right that they tried Makefile.PL first. However, in the final analysis, whether they tried it first or not is immaterial. So long as Module::Build is available to try at all, it means that a makefile backed by Module::Build will work.

        ...I am referring to the contents of @INC before any script calls use lib or any command line sets and exports PERL5LIB.

        I'm not saying this is the recommended way to do it, but technically, the following should work:

        Set PERL5LIB in the Makefile

        export PERL5LIB = ...path/to/lib

        As any command being run for the "test" target (or any other target) is a subprocess of make, it will inherit the environment variable, so the lib dir is being added to @INC for all tests.

        What remains to be solved is how to get MakeMaker to put the above line in the generated Makefile. A quick test shows that this can be achieved by overriding the routine post_initialize, which is responsible for creating a (normally empty) section at the beginning of the Makefile.  I.e., add a MY::post_initialize routine to your Makefile.PL:

        use ExtUtils::MakeMaker; sub MY::post_initialize { return "export PERL5LIB = ...path/to/lib"; } WriteMakefile( ... );

        This generates a section in the Makefile

        # --- MakeMaker post_initialize section: export PERL5LIB = ...path/to/lib

        which causes make to set PERL5LIB in its environment.

        ExtUtils::Command::MM::test_harness adds $(INST_LIB), i.e. blib/lib, and $(INST_ARCHLIB), i.e. blib/arch to @INC. It has no provision for adding anything more to @INC

        Exactly. Your claim that there's something special about the project root is completely false.

        I simply don't buy that it is a "good thing" to go around coding "use lib" statements in every single .t file.

        It's a necessary thing, not just in .t, but in all scripts that use uninstalled modules. If you don't, Perl won't find them. If the module isn't somewhere where Perl would normally look (e.g. the directory in which the script resides), you need to tell Perl where it is.

        No idea why you bring up CPAN and legal protections.

        First, it means that I have to go into every file I want to ship and edit it to have that line.

        No, just the scripts.

        Secondly, these support libraries are used in a lot of different projects

        Straw man. If Perl can find your libraries, they don't need to tell Perl where to find your libraries.

        Secondly, if you have a module that used by multiple projects, it's silly to copy it into each project and adjusting @INC to find that copy. Install the module.

        I have no need of it when I let Module::Build build the project.

        So you're proudly claiming that one needs to use a giant harness to get your test script to work properly, ignoring that it fails the rest of the time.

        I am not proposing doing away with Makefile.PL.

        I didn't even address your proposal. I just corrected the false claim that only 3% of testers try to run Makefile.PL.

        To address the proposal: It's a lot of work and you claim introduces problems, while you can get the desired effect by simply adding inc/ to the harness command. And your test script still won't run outside the harness.

        Before you reply that you don't expect your test scripts outside of Module::Build, realise there's no problem if that's true.

Re: Do Pure Perl CPAN packages really need to use ExtUtils::Command::MM?
by Anonymous Monk on Feb 18, 2011 at 02:02 UTC

      Thanks. However, that is essentially option A: bite the bullet. It would have to go in every single test file and every single set up script. If I am going to use ExtUtils::Command::MM to back my makefile, I either need

      • A compelling argument for why a makefile backed by Module::Build is not good enough. There is no need for Find::Bin+use lib when Module::Build backs the makefile. If it really is not good enough, I would be willing to bite the bullet.
      • A solution that does not require me to hardcode Find::Bin + use lib in file after file. Eliya has proposed one solution but there are still some outstanding portability questions.
        If I am going to use ExtUtils::Command::MM to back my makefile, I either need: A compelling argument for why a makefile backed by Module::Build is not good enough.

        Hmm, so you're writing your own Makefile (no .PL)? That is the worst of all solutions :) --Module::Build::Compat (see Module::Build users -- please use the "traditional" create_makefile_pl option) ++Module::Install

        A solution that does not require me to hardcode Find::Bin + use lib in file after file. Eliya has proposed one solution but there are still some outstanding portability questions

        Why is that a painful/unpleasant/unavoidable solution? You only have to do it once. You don't even have to do it manually :) You can make it part of your template or automate it

        There is even use inc 'Plagger'; which you can use outside of Module::Install

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://888753]
Approved by jettero
Front-paged by Old_Gray_Bear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (5)
As of 2024-03-19 08:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found