MakeMaker-Makefile Reform School

[Intrepid]

Reforming the Makefile Madness for my specific machine / OS

NOTE: Updated Sat Apr 26 10:43:42 UTC 2008 to fix broken links pointed out in child node 682635

make and Makefiles and all pertaining to that are so uncool that for me they've wrapped right around back into cool.

This node documents what I have invented to serve the end purpose of having builds of perl modules go more smoothly (well, builds either "go" or "not go", so let's say "smoothly" == "to proper completion") on my non-Unix system. The description of that system follows below, but for now let's skip forward to avoid bad bogged-downnesses.

The old standard for creating a Perl module (or other) distribution is to create a file Makefile.PL in the toplevel of the distro tree. I'm not going to discuss Module::Build in this node (or its children) because I do not use it and am interested in (a fan of) GNU make. So this writeup is about the efficient use of the generated Makefile that we'll have after running the classic first-step of the module-building invocation triplet (perl Makefile.PL; make test; make install). It's not about other theoretical or half-realized approaches to installing perl modules.

The Makefile that is generated by MakeMaker.pm's WriteMakefile function is a big water buffalo, there is no debate about that from this corner. This does not make it impossible to comprehend, however. Yours Truly has spent many hours poring over Makefiles for many packages, both perl and non-perl in nature, and can say this much: there is a method (heheh) to the madness. Unlike what the Camel says about trying to read the output of some of the Perl-to-C generators, one will not go blind trying to read this generated Makefile. One may well want to beat one's dogs afterwards, but one will not go blind!

Before going much further the author feels obligated to allow the reader to sample what a classic perl MakeMaker-generated Makefile looks like. So one is provided for inspection here. One was pulled at random out of the .cpan/build directory. It's to the Data::Dumper package, which does not (in the authors opinion) sin particularly in regards to MakeMaker.pm misuse.

The woes that can afflict the user (installer, admin or otherwise) of this system can include having a Makefile generated which does not give correct or complete directives for telling the C tools (cc, ld) how to compile and link the C code part of the module. On the particular system in question this problem was ameliorated in part by a completely separate scheme which will someday be described in greater detail elsewhere, but which in brief, uses the interesting package ExtUtils::FakeConfig (not CORE and not carried on PPM in its up to date revision, at time of this writing) to override some of the key %Config::Config hash values. By replacing the ones shipped by a vendor (which were supposedly correct for the system on which the Perl was built) with ones correct for the system on which Perl was installed (and on which the attempt is being made to install additional modules), this useful tool has allowed the author to rather painlessly add quite complex modules to his Perl installation.

It being high time to describe this "non-Unix" system, it shall be done: it is a ActiveState Perl downloaded and installed in the standard manner (standard for AS), and the one used for development was specifically perl 5.8.0, AS build 806. The system is Windows XP. The twist is that the build environment is not similar to the one which most users familiar with Perl on Win32 expect to be limited to:

• The build shell used for entering the commands to install the module from distro src is (cygwin-provided) bash, a POSIX-conformant (basically) Unix shell;
• The compiler for C/C++ code is MinGW port of GCC 3.4.2;
• The binutils accompanying that GCC are those that come in the same MinGW release (in this case, as provided in MinGW "3.20 rc3", a pre-release).
• The make "utility" (a poor label for such a capable and brilliant tool) is GNU make, not the vastly inferior (in the author's opinion) and unfree (and unspellable) nmake from M$.

Here are a few issues that such a system causes to arise:

1. The paths to files, both tools and dependencies, are generated with and derive from pathnames that use the backslash \ as the path component delimiter, instead of the perl-standard forward slash. Since we are using a Unix shell, this is not going to work out: to a Unix shell the backslash is a character laden with magical properties, e.g., it is used to escape what follows it. If it is followed by another backslash then its magicalness is neutralized, but we've just created a thicket of backslashes which displeases the author on every level (practicality - impairs readability; esthetics - looks like hell; expediency - makes it harder to spot possibly erronious parameters).
   
2. The syntax of the rules used to run the C compiler (cc, gcc) is not always completely right for modern versions of GCC. The stuff in ExtUtils::MakeMaker that generates this Makefile rule code is ancient, hailing from a gcc v1 era, and badly in need of reform. Even when it works it does so in a "fragile" way, not saying what we need to say to the master cc driver in the most concise and efficient manner. Since I <heart> my C compiler, I don't want it treated that way ;-)

We can use the fact that Makefile.PL is just a Perl program to create better, prettier and more accurate Makefiles by placing overrides for some of these things into the Makefile.PL. How to override the overridable MakeMaker methods is fully documented in other places (such as the manpage for MakeMaker) so it will not be elicudated extensively here. But item (1) in the list above is a thornier problem: there is no global method or tactic for making all those perverted backslashes lean in the correct direction ;-), and that's a showstopper if we do not repair it. The more severe means that author has devised rewrite the entire Makefile (and incidentally output it under a different name: GNUmakefile) with several types of mutations performed on it. The slash problem is corrected this way.

Because GNU make when invoked with no -f (as we customarily do it in the perl world) prefers GNUmakefile to plain "Makefile" or "makefile", our reformulated input to make is what gets used to build, test and install the module package's files. The tasks that need to be accomplished to cause perl Makefile.PL to generate this nicer GNUmakefile for any given module package should not be too burdensome for the author, or he won't do it (Perl impatience: "don't make me repeat lengthy procedures in duplicate instances"). What has worked so far is described next.

There are some things which the author needs to tell Perl about before the WriteMakefile() subroutine is invoked in the Makefile.PL, and there are some things which much take place after that WriteMakefile() has been invoked and created the Makefile. To accomplish this any module's Makefile.PL can just have two alterations made to it at appropriate locations in the file (before and after). These alterations are the addition of a single line in each case: a perl require() statement. Here's an example, for the CPAN module C::Scan:

use ExtUtils::MakeMaker;

require "C:/DOCUME~1/intrepid/APPLIC~1/ActiveState/PerlWin32MakefilePL
+begin";

# See lib/ExtUtils/MakeMaker.pm for details of how to influence
# the contents of the Makefile that is written.
WriteMakefile(
    'NAME'      => 'C::Scan',
    'dist'      => { COMPRESS => 'gzip', SUFFIX => '.gz'},
    'VERSION_FROM' => 'Scan.pm', # finds $VERSION
    'LIBS'      => [''],   # e.g., '-lm' 
    'DEFINE'    => '',     # e.g., '-DHAVE_SOMETHING' 
    'INC'       => '',     # e.g., '-I/usr/include/other' 
    'PREREQ_PM' => {'Data::Flow' => 0.05},
);

require "C:/DOCUME~1/intrepid/APPLIC~1/ActiveState/PerlWin32MakefilePL
+finish";

There are two require statements that don't exist in the distributed (CPAN/upstream) unaltered Makefile.PL file. These "pull in" perl code at the right points in the flow of compilation and execution. That perl code is contained in two files which can be viewed at the links below. By keeping them in a common location their reuse is facilitated. Other, better ways of making this possible may yet be devised by the author. This is a work-in-progress. YMMV. Caveat Scriptor.

• The opening code (pre- WriteMakefile).
• The finalizing code (post- WriteMakefile).

---

Disclamers and Notices

There is every likelihood that the author will be needing to make revisions to this node after its initial posting.

No debates about Module::Build please. And please try to take debates about the merits of using "external tools like make" to other threads, as well. These areas have all been thoroughly discussed in the past. Difference in opinion exist. While the author relishes a rousing flamewar as much as the next Monk, he will consider raising those issue on this node a form of defacement or vandalism or at minimum, deep disrespect.

The ExtUtils::FakeConfig package provides a "template-driven" way to override Perl buid parameters. The package provides starting points for the MinGW tools, but I found that I needed to add and make changes to the provided Config_m file. My changes as they currently exist may be seen here.

---

May the Force be with you. Subvert the Dark Side to noble ends!     ;-)

[Joost]

How to override the overridable MakeMaker methods is fully documented in other places (such as the manpage for MakeMaker)

I've looked over the ExtUtils::MakeMaker manpage several times, but I still can't find any documentation on proper use of its (to my eyes) completely weird &MY::stuff type of "inheritance" (or whatever it is).

Makemaker is still one of perl's biggest mysteries to me, so if you have any hints on where to find this documentation, please please please share it.

update: As an example, the docs say

       [...] you can edit the default by saying something like:

               package MY; # so that "SUPER" works right
               sub c_o {
                   my $inherited = shift->SUPER::c_o(@_);
                   $inherited =~ s/old text/new text/;
                   $inherited;
               }

And then remains silent on what the &c_o method is supposed to do, and only mentions &postamble, which doesn't do anything by default. Likewise, there is no mention of any method to override to create your own META.yml file (and there is one, but I forgot what it was).

update2: fixed broken arrow...

J.

[Intrepid]

Joost wrote

I've looked over the ExtUtils::MakeMaker manpage several times, but I still can't find any documentation on proper use of its (to my eyes) completely weird &MY::stuff type of "inheritance" (or whatever it is).

---

Note: within 15 minutes or so of posting this reply, I had to fix some typos and errors in the example at bottom. Should read correctly now.
Update Notice: (2005.04.03) - I've realized based on some recent discussions that the documentation cited is probably ineffectual / wrong ... so the "expanded" sub-method override code at the bottom of this message now shows us using a diferent means than "::SUPER" to accomplish what's needed.

---

It takes some getting used to. I'll try to help with a little elucidation of what happens, etc. OO terminology not being my very strongest point, it may be that others will offer pointers too.

The direct answer to your original question is: the documentation in the POD for ExtUtils::MakeMaker that you pointed to is the Fun{damental} Documentation. My 2Ed Camel ("Programming Perl" for newbies ;-) gives the same basic example and text. There is no other.

The answer to the implied question is this:


• Look at the build of the module as it proceeds. Is anything amiss? Read every line of the compiler output. If nothing is broken, there's nothing to fix.
  
  
• If you find that there's something to fix, then that something lives in the Makefile. Each MM method that outputs text leaves a snailtrail that looks like this, in the Makefile:
  

       # --- MakeMaker <mm_methodname> section:
  

  where mm_methodname happens to be the name of the MM method that creates the text that follows. All you have to do is "reverse-engineer" by finding the part of the Makefile where things are not going as you wish (make conforms to the MJD principle just like any other computer programming: it cannot magically guess what you need, you have to tell it in the proper manner). Once you have done that, you know what method you want to override.

A little warning here:

Although it probably should not be, there is some interdependency between the MM methods that are overridable. I have found that overriding one made output of another break. This is not, I think, in conformance with the original intentions of the early MM authors, and may be bugs introduced in the later rewriting which has been going on.

Back to our show

• ... the method override is written into the Makefile.PL. It does not matter whether it comes before the WriteMakefile() invocation or not: it is a sub declaration, it is grokked at compile time.

Now that we've gotten the "why" or "when" (might I want to override a MM method) out of the way, and touched on the "how", let's go back and look at the "ehhhh?!?".

Understand that this IS OO programming. When use ExtUtils::MakeMaker; is seen, there's going to be an implicit instantiation of a ExtUtils::MakeMaker object happening. Right away. By perl mechanism this object is the first arg passed into all method calls; by MakeMaker convention the object is assigned to the scoped lexical var $self. Thus we are always operating on a $self unless we choose to just throw away the object, as in:

sub MY::makefile    { return "# removed" }

In this example I have left a little snail slime (a comment) in the Makefile, where the "stuff" that once directed the re-making of the Makefile once was. I ignored the MM object completely. The point here is that the MM methods which exist to create segments of the Makefile just output a string; this is quite simple actually. Each is like a part of a modern modular assembly-line process for creating manufactured goods, like automobiles. MakeMaker takes care of assembling all the parts into the final vehicle (the output Makefile). You can modify any part by either taking the "stock" part and working modifications on it, or chuck it in the junk heap completely and start from scratch. "Monster Garage" has nothing on us!

The example given in a recent ExtUtils::MakeMaker POD page shows both approaches. Their example of "modification of the stock part" looks like this:

package MY;
        sub c_o {
            my $inherited = shift->SUPER::c_o(@_);
            $inherited =~ s/old text/new text/;
            $inherited;
        }

This (bogus) example shows overriding of the MM method that generates the Makefile entries for remaking object files from C code modules.

Saying package MY; sub c_o .... is just the same as saying sub MY::c-o { ..... There is no difference whatsoever.

This is inheritance (Perl-style) at work, yes. But one doesn't really have to understand why or how it works in order to use it, one can just take the example (and hours of poring over the POD and source for ExtUtils::MM_Unix) ;-) and modify it to suit one's requirements. The bottom line is that anything declared into the MY:: package is going to have the final word on what gets placed in the Makefile.

Rewritten to remove some terse obfuscating confusion, the above example could look like this (commented to aid understanding):

 sub MY::c_o {
     my $self = shift( @_ );  # get our MM object
     local *c_o;              # localize to inside our current scope a
+t runtime.
     my $inherited_text = $self->MM::c_o( @_ ); # any other args get p
+assed to
                                                # the "stock" MM metho
+d (there
                                                # generally are none).
     $inherited_text =~ s/old text/new text/; # do vengeance on those 
+who have
                                              # wronged us.
     return $inherited_text; # a return of a scalar whose value is jus
+t a
                             # (probably multi-line) string.
 }

HTH!

[ysth]

The ExtUtils::MakeMaker documentation directs you to ExtUtils::MM_Unix for documentation of all MakeMaker methods, where it says:

c_o (o)
    Defines the suffix rules to compile different flavors of C files to object files.

To see what that looks like, open up a generated Makefile and look for "MakeMaker c_o section:".

[bart]

To accomplish this any module's Makefile.PL can just have two alterations made to it at appropriate locations in the file (before and after). These alterations are the addition of a single line in each case: a perl require() statement.

I wonder if there isn't a way to do it without altering the original Makefile.PL. I think 'd write a module, say GNUMakefile.pm, which you could invoke from the command line like

perl -MGNUMakefile Makefile.PL

Now, I'm still sketchy on the details on what it should do. It should just run your "begin" section as it loads, so just including the contents of that file ought to be enough. As for the finalization code, that should execute when the Makefile is written, so perhaps call its main action in an END block, in the module, might work?

Alternatively, I'd think of a source filter, virtually modifying the Makefile.

Yet another option is adding (pre- and) post-hooks, to be called after WriteMakeFile() has finised, for example with Hook::LexWrap.

Note that these are just ideas, I haven't tested any of these approaches to see if they actually fly.

[rkeefe]

Intrepid, I read you post on fixing Makemaker by adding the 'require' statements around a module's Makefile.PL. It seems that the links are broken for your code examples. Are these still available? Thanks.

[Intrepid]

rkeefe wrote:

It seems that the links are broken for your code examples. Are these still available? Thanks.

Unfortunately those files are were no longer at the URLs pointed to because of the widely-known breakdown of the perlmonk.org server. Compounding that misfortune, yours truly has been "too busy" to go back and make sure that files from the old account are back in place.

I'll try to fix those links ASAP. Thank you very much for your interest in Makefile Reform :-).

Update: fixed URLs / file locations for the 2 pieces ("Before" and "After") [Sat Apr 26 10:43:42 UTC 2008]

[syphilis]

Intrepid, I read you post on fixing Makemaker by adding the 'require' statements around a module's Makefile.PL

I'm a little surprised that you feel the need to pursue that approach. If you're using Gnu make and/or a Unix-type shell, then the links you seek may well contain useful information for you. But if you're using the freely available dmake (or nmake) in the cmd.exe shell then MakeMaker shouldn't need any fixing at all.

I have used ExtUtils::FakeConfig with ActivePerl build 806 to build a number of extensions - and have not experienced any need to fix MakeMaker.

Cheers,
Rob